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Our  Job  Is  to  Get  It  There 

One  of  the  beauties  of  software  is  how  quickly  we  can  make  changes  compared  to  the 
hardware  world.  This  has  given  rise  to  the  tremendous  expansion  of  military  soft* 
ware  applications,  which  has  occurred  during  the  past  thirty  odd  years.  I  recently  saw  an 
illustration  that  plotted  the  number  of  lines  of  software  on  U.S.  fighter  aircraft  starting 
with  the  F-lll  and  F-4  and  progressing  to  the  F/A-22  and  F-35.  It  was  exponential  as 
one  might  expect  (we  software  types  are  beneficiaries  of  Moore’s  Law  and  hardware 
improvements  after  all).  The  interesting  point  to  remember  is  that  this  embarrassment  of 
riches  contains  a  daunting  challenge,  namely  efficiently  harnessing  this  great  flexibility  that  software 
affords.  We  have  found  over  the  years  that  there  is  “no  royal  road  to  learning”  as  Euclid  said.  To 
do  software  right  we  have  to  do  it  the  old-fashioned  way,  that  is,  relentless  commitment  to  quality: 
employing  peer  reviews,  configuration  control,  documentation,  and  testing. 

The  topic  of  this  issue  is  “Systems:  Fielding  Capabilities.”  It  is  really  the  reason  that  all  of  us 
who  provide  software  capability  are  employed.  It  is  our  job  to  get  what  is  needed  to  those  who  need 
to  use  it.  Often  we  think  of  software  as  an  intellectual  pursuit.  All  of  us  have  felt  the  exultation  of 
solving  a  difficult  problem  or  finding  a  creative  solution  but  ultimately  we  must  focus  on  our  cus¬ 
tomer,  not  ourselves.  To  this  end,  it  is  not  enough  that  we  satisfy  ourselves  with  our  software  abil¬ 
ities  but  that  we  satisfy  those  who  depend  upon  us.  Fielding  capabilities  entails  doing  our  work  both 
well  and  quickly.  Well  so  the  warfighter  is  not  disappointed,  or  worse,  in  undue  risk,  and  quickly  so 
the  warfighter  is  not  kept  waiting.  This  issue  will  help  you  do  that. 


Thomas  F.  Christian,  Jr. 

Warner  Robins  Air  Logistics  Center  Co-Sponsor 
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From  the  Publisher 


Stay  Focused  on  the  User 


This  month’s  theme,  “Systems:  Fielding  Capabilities,”  developed  during  a  discussion  with 
CROSSTALK’S  sponsors  about  merging  hardware  and  software.  As  they  considered  this, 
they  emphasized  that  importance  should  not  be  placed  on  the  system  components  of  hard¬ 
ware  and  software;  importance  should  be  placed  on  the  capability  required  by  the  user.  This 
is  such  an  important  concept  that  we  decided  to  build  a  theme  around  it. 

We  start  our  theme  discussion  with  Key  Llements  in  Fielding  Capabilities  by  John  D.  Holcomb 
and  Michael  Hoehn.  These  authors  discuss  the  benefits  of  testing  environments  that  emu¬ 
late  the  true  user  environment  and  the  need  for  ongoing  communication  among  stakeholders.  In 
Delivering  Capabilities  Through  Fartnerships,  Chris  D.  Moore  details  his  experience  with  a  military-industry 
partnership  focused  on  providing  the  U.S.  military  with  needed  core  competencies  to  ensure  contin¬ 
ued  support  to  the  warfighter.  We  conclude  our  theme  articles  with  MILS:  Architecture  for  High  Assurance 
Embedded  Confuting  by  several  authors  who  discuss  information  assurance  as  a  method  for  enabling  the 
capability  to  win  wars  via  superior  knowledge. 

We  continue  this  issue  with  our  supporting  articles.  In  Six  Steps  to  a  Successful  COTS  Implementation, 
Arlene  F.  Minkiewicz  shares  key  points  to  contemplate  and  implement  when  considering  commercial 
off-the-shelf  as  part  of  your  end-product.  Paul  J.  Solomon  and  Bill  Ravensberg  also  contribute  with 
Performance-Based  Earned  Value  and  Balanced  Scorecards:  From  Golf  to  Business ,  respectively,  explaining  how 
earned  value  and  balanced  scorecards  are  each  key  to  tracking  project  progress. 

CROSSTALK  continues  to  aim  to  provide  readers  with  the  capability  to  buy  and  build  software  bet¬ 
ter.  I  hope  we  further  our  mission  with  this  issue. 


Elizabeth  Starrett 
Associate  Publisher 
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Key  Elements  in  Fielding  Capabilities 

John  D.  Holcomb  and  Michael  Hoehn 

76th  Softivare  Maintenance  Group 

Organisations  that  develop  softivare  for  the  Department  of  Defense  must  have  knowledgeable  people  to  do  the  work  accord¬ 
ing  to  documented  and  mature  processes  and  standards  that  guide  how  the  work  is  accomplished.  The  organisation  must  also 
have  in  place  the  hardware  and  tools  that  are  used  to  execute  the  processes  to  develop  and  test  the  end  products.  Success  of 
their  efforts  depends  on  two  key  elements:  the  fidelity  of  the  test  environment,  and  the  amount  of  collaboration  with  other  agen¬ 
cies  involved  in  their  program. 


The  mission  of  the  76th  Software 
Maintenance  Group  (SMXG,  former¬ 
ly  MAS)  at  the  Oklahoma  City-Air 
Logistics  Center,  Tinker  Air  Force  Base 
(AFB),  Okla.,  includes  positioning  opera¬ 
tional  capabilities  in  the  field,  and  improv¬ 
ing  and  adding  to  them  through  software 
development  and  sustainment.  The  76th 
SMXG  performs  this  mission  for  the  E-3, 
B-l,  B-2,  and  B-52  aircraft,  and  for  the  Air 
Launched  Cruise  Missile  (ALCM), 
Conventional  Air  Launched  Cruise  Missile 
(CALCM),  and  Advanced  Cruise  Missile 
(ACM)  weapons.  The  group  also  has 
extensive  capability  for  development  and 
maintenance  of  Test  Program  Set  hard¬ 
ware  and  software  for  automatic  test 
equipment;  industrial  automation;  and  jet 
engine  testing,  trending,  and  diagnostics. 

Any  organization  that  develops  or  main¬ 
tains  weapon  system  software  must  have 
certain  resources  in  place.  These  resources 
include  people,  a  development  environ¬ 
ment,  a  test  environment,  tools,  and  facilities 
to  house  these  resources.  Policies,  instruc¬ 
tions,  standards,  and  processes  are  required 
to  control  how  the  work  is  accomplished. 
Measurement  and  metrics  requirements 
must  be  established  to  facilitate  tracking 
workload/labor;  to  evaluate  financial  and 
project  performance;  and  to  establish  the 
foundations  for  pricing  future  projects, 
making  management  decisions,  and  process 
improvement  efforts.  We  have  an  outstand¬ 
ing  Software  Engineering  Process  Group 
(SEPG)  that  organizes  and  develops 
processes  and  standards,  and  maintains 
them  online.  It  also  establishes  and  manages 
our  measurement  and  metrics  requirements 
and  process  improvement  efforts  and  the 
organizational  software  quality  program. 

The  76th  SMXG  consists  of  approxi¬ 
mately  500  engineers,  computer  scientists, 
and  staff  personnel,  the  majority  of  whom 
have  in  excess  of  15  years  experience  in  the 
organization.  Various  development  environ¬ 
ments  are  used  based  on  the  weapon  system 
or  automatic  test  equipment  that  the  appli¬ 


cation  software  runs  on,  however,  most 
applications  are  developed  on  IBM  main¬ 
frames,  Sun  workstations,  or  networked  per¬ 
sonal  computers.  Tools  used  include  assem¬ 
blers,  compilers,  and  tools  for  project  plan¬ 
ning  and  management,  labor  tracking, 
requirements  tracking,  configuration  man¬ 
agement,  documentation,  etc.  A  detailed  dis¬ 
cussion  of  all  aspects  of  our  operation  is 
not  possible  within  the  scope  of  this  article, 
and  most  readers  are  aware  of  these  aspects 
from  their  own  experience.  Thus  this  article 
will  focus  on  two  key  areas  that  reduce  risk 
when  fielding  operational  capabilities:  high 
fidelity  test  environments  and  collaboration. 

High  Fidelity  Test  Environments 

The  test  environment  is  one  of  the  key  ele¬ 
ments  in  fielding  capabilities.  If  it  does  not 
emulate  the  fielded  system  to  the  maximum 
extent  possible,  then  the  risk  of  operational 
problems  when  the  software  is  fielded 
increases.  The  initial  Avionics  Integrated 
Support  Facility  Military  Construction 
Project  at  Tinker  AFB  was  built  in  the  early 
1980s  to  provide  floor  space  to  house  the 
software  support  personnel  and  develop¬ 
ment  environments  for  the  E-3,  B-52,  Short 
Range  Attack  Missile  (SRAM),  and  the 
ALCM.  An  example  of  our  high  fidelity  test 
environments,  the  B-52  Avionics  Integrated 
Support  Facility  (AISF),  is  a  hot  mockup  of 
the  aircraft  avionics  interfaced  with  the  con¬ 
trols  and  displays.  Simulated  dynamics  of 
the  aircraft  are  provided  by  a  vehicle  system 
simulator,  and  weapon  simulation  is  provid¬ 
ed  by  a  weapon  system  simulator. 

The  SRAM  and  ALCM  laboratory  area 
was  built  adjacent  to  the  B-52  AISF  area. 
The  cruise  missile  project  provided  inter¬ 
faces  between  the  B-52  AISF  and 
empty/loaded  pylon/launcher  station  as 
well  as  between  the  AISF  and  the  ALCM 
subsystem  simulator.  This  test  environment 
is  used  for  simulation  and  test  of  the  B-52 
operational  flight  software,  the  aircraft  to 
missiles  interfaces,  and  the  missile  opera¬ 
tional  flight  software.  By  utilizing  the  com¬ 


bination  of  the  B-52  AISF  and  pylon  or 
launcher  loaded  with  test  missiles,  all  com¬ 
munication  between  the  aircraft  and  missiles 
can  be  effectively  tested.  The  SRAM  pro¬ 
gram  has  been  disposed  of  and  the  missiles 
laboratory  has  evolved  through  the  years  to 
include  capability  for  ACM  and  CALCM. 

A  missile  electronic  subsystem  simula¬ 
tor,  consisting  of  a  table  of  interfaced  mis¬ 
sile  electronics,  is  also  available.  Breakout 
boxes  can  be  used  at  the  umbilical  connec¬ 
tor,  or  at  any  of  the  internal  missile  interface 
connectors  to  allow  monitoring  of  the  inter¬ 
faces.  Testing  of  the  B-52  operational  flight 
software  and  missile  operational  flight  pro¬ 
gram  is  accomplished  by  first  planning  the 
mission,  then  flying  the  aircraft  mission  on 
the  AISF,  rotating  the  launcher  to  the  prop¬ 
er  missile,  if  necessary,  then  simulating 
launch,  and  finally  simulating  free  flight  of 
the  missile  to  target. 

The  AISF  interface  with  the  ALCM  sub¬ 
system  simulator  is  used  to  test  aircraft/mis¬ 
sile  interface  up  to  launch  and  subsequently 
test  free-flight  simulation  of  the  missile 
operational  flight  program  from  launch  to 
detonation  at  target.  Successful  testing  of  B- 
52  and  missile  operational  flight  software 
and  mission  planning  software  in  this  labo¬ 
ratory  provides  very  high  confidence  that 
flight  testing  will  be  successful  and  that  the 
software  will  provide  the  required  capability 
when  fielded. 

A  government  owned  and  operated  test 
environment  for  weapon  systems  has  bene¬ 
fits  other  than  the  ability  to  fully  test  the 
software.  Having  this  capability  in  a  govern¬ 
ment  facility  allows  it  to  be  used  for  com¬ 
petitive  procurement  of  projects  that  are 
beyond  organic  capability.  For  example,  a  B- 
52  modification  was  programmed  and  fund¬ 
ed,  but  the  sole-source  contractor’s  price  for 
developing  the  modification  at  the  compa¬ 
ny’s  facility  was  in  excess  of  the  budget.  We 
recommended  to  the  program  office  that 
the  project  be  competitively  procured  based 
on  performance  in  our  government  facility. 
The  effort  was  competed,  development  and 
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testing  was  accomplished  in  die  AISF  by  the 
winning  contractor,  and  the  final  cost  was 
approximately  one  half  of  die  original  bid. 

Another  additional  benefit  is  the  expert¬ 
ise  that  organic  personnel  develop  as  a  result 
of  having  this  type  of  laboratory.  An  exam¬ 
ple  of  this  occurred  after  two  B-52/ALCM- 
W80-1  Joint  Test  Assembly  (JTA)  flight 
tests  resulted  in  aborted  launches  with  total 
mission  failure.  An  analysis  of  the  mission 
data  indicated  a  problem  between  the  B-52 
offensive  avionics  system  (OAS)  and  the 
missile  test  payload  (W80-1  JTA).  After 
returning  to  the  home  base,  the  aircraft 
underwent  extensive  ground  testing  by  Air 
Force  personnel  and  no  problem  could  be 
found.  Sandia  National  Laboratories  per¬ 
formed  a  series  of  comprehensive  tests  on 
their  JTA  package  and  concluded  that  it  did 
not  contribute  to  die  aborted  launches. 
Sandia  further  prepared  a  letter  to  the 
Cruise  Missile  Product  Division  detailing 
their  findings  and  recommending  the  Air 
Force  suspend  future  B-52/ALCM  JTA 
flight  tests  until  the  Air  Force  could  identify 
die  source  and  correct  the  problem. 

This  problem  was  referred  to  our  engi¬ 
neers,  and  the  B-52/missiles  laboratories 
were  configured  for  an  ALCM  JTA  launch 
using  missile  and  B-52  production  hardware 
and  operational  software.  Utilizing  state-of- 
the-art  recording  and  analysis  tools,  engi¬ 
neers  performed  multiple  JTA  launches. 
Analysis  of  the  laboratory  flight  test  data 
and  the  JTA  data  from  the  aborted  launches 
clearly  determined  that  die  aborted  launches 
were  a  result  of  the  JTA  package.  Flaving 
determined  die  source  of  die  problem,  engi¬ 
neers  from  the  AISF  presented  dieir  find¬ 
ings  identifying  the  JTA  as  die  source  of  the 
problem  and  isolating  specific  circuits  in  the 
JTA  diat  were  suspect.  Sandia  accepted 
tiiese  findings,  had  additional  testing  per¬ 
formed  on  the  suspected  circuits,  and  was 
able  to  isolate  the  specific  failure  mode. 

Similar  test  environment  capabilities 
exist  in  all  of  our  weapon  system  laborato¬ 
ries.  For  example,  the  E-3  Airborne 
Warning  and  Control  System  (AWACS)  lab¬ 
oratory  has  bodi  surveillance  radar  configu¬ 
rations;  this  way,  all  versions  of  the  E-3  soft¬ 
ware  can  be  tested.  During  Desert 
Shield/Desert  Storm,  an  enhancement  was 
required  to  support  that  effort.  The  require¬ 
ment  was  identified  on  Thursday.  The 
change  was  programmed,  implemented, 
tested  in  die  laboratory,  flight  tested  at 
Tinker  AFB,  and  sent  to  the  theatre  on  a 
resupply  flight  the  next  Monday.  This 
demonstrated  die  fast  turnaround  capabili¬ 
ties  of  organic  resources  widi  high  fidelity 
test  environments. 

This  system  has  also  helped  with  soft¬ 
ware  not  developed  by  our  organization. 


During  die  early  1980s,  die  AWACS  wing 
experienced  a  serious  problem  with  the  E-3 
navigational  computer  system.  The  system 
consistendy  failed  to  capture  the  turn  por¬ 
tion  of  an  established  surveillance  orbit, 
potentially  causing  die  E-3  to  fly  into  unau- 
diorized  airspace.  Serious  consequences 
were  narrowly  avoided  on  several  occasions. 
Once  notified  of  the  problem,  the  E-3  AISF 
was  able  to  reproduce  die  problem  in  the 
laboratory,  locate  the  source  in  anodier 
organization’s  code,  and  develop  a  fix.  The 
modified  operational  flight  program  was 
tiien  tested  in  die  E-3  AISF  and  delivered  to 
die  E-3  fleet  in  a  timely  manner. 

These  types  of  weapons  system  test 
environments  are  normally  established  as  a 
part  of  the  weapon  system  development 
program  at  die  prime  contractor’s  facility. 
The  test  equipment  is  dien  either  duplicated 
at  or  transferred  to  a  government  facility  for 
support  of  the  weapon  system  software 
after  deployment  of  the  weapon  system. 
Since  the  test  environment  includes  the 
avionics  suite,  the  cost  is  very  high.  In  1995, 
replacement  cost  estimates  for  the  AISFs 
were  as  follows: 

•  B  1 :  $171  million. 

•  B-52:  $54  million. 

•  Missiles:  $51  million. 

•  E-3:  $100  million. 

Each  AISF  is  unique  and  may  have  more 
dian  one  hot  mockup  of  die  avionics. 
Simulation  computers  are  typically  Harris 
(now  Concurrent)  computers  and  use 
Fortran,  C,  or  C++,  as  the  source  lan¬ 
guage^)  for  the  simulation  software 
depending  on  die  specific  application.  Our 
experience  is  that  die  high  cost  of  these 
high  fidelity  test  environments  is  well  wordi 
the  investment  because  diey  enable  the  soft¬ 
ware  support  activity  to  provide  die  cus¬ 
tomer  and  the  warfighter  with  die  required 
capabilities  when  diey  are  needed. 

Collaboration 

Another  key  element  in  fielding  capabilities 
is  teamwork  between  the  program  manager, 
system  engineer,  software  developer, 
warfighter,  and  tester.  The  B-52  Mission 
Planning  Software  Section  is  one  of  our  top 
success  stories.  This  section  has  responsibil¬ 
ity  for  development  and  sustainment  of  the 
B-52  mission  planning  software  (B-52 
Aircraft/Weapons/Electronics  [A/W/E]) 
that  runs  on  the  Air  Force  mission  support 
system  (AFMSS).  This  system  essentially 
automates  the  process  that  the  flight  crews 
previously  performed  manually  —  according 
to  the  Technical  Orders  (TOs),  which  are 
used  as  the  basis  for  the  software  require¬ 
ments.  One  of  die  most  important  keys  to 
success  is  customer  involvement.  To  ensure 
diat  the  product  produced  meets  the 


user/customer  requirements,  die  warfight¬ 
ers  are  involved  in  each  phase  throughout 
the  development. 

The  B-52  A/W/E  is  one  element  of  the 
complete  mission  planning  environment, 
which  is  a  combination  of  35  different  ele¬ 
ments  of  software  and  17  pieces  of  dedicat¬ 
ed  B-52  aircraft  software,  as  shown  in  Figure 
1  (see  page  6);  the  Glossary  defines  the 
terms  in  die  figure.  These  are  developed  by 
different  agencies  and  contractors,  and 
many  serve  multiple  weapon  systems. 
Integration  of  all  of  these  applications  on  a 
single  system  to  meet  overall  warfighter 
requirements  is  a  significant  part  of  our 
effort. 

The  main  key  to  success  in  this  area  is 
the  in-depth  understanding  of  die  entire 
environment,  botii  hardware  and  software. 
The  mission  planning  process  is  tested  from 
beginning  to  end,  including  the  production 
of  all  flight  products.  These  products  are 
taken  through  the  final  verification  of  actu¬ 
ally  loading  diem  in  our  B-52  AISF  and  fly¬ 
ing  the  missions,  complete  with  weapons, 
and  the  recording  of  all  the  data  for  analysis. 

Ensuring  success  begins  widi  customer 
or  user  relationships.  More  than  a  tiiird  of 
our  key  people  who  develop  and  maintain 
mission  planning  applications  are  on  a  first- 
name  basis  with  dozens  of  stakeholders: 

•  B-52  System  Program  Office.  All  mis¬ 
sion  planning  system  engineers,  pro¬ 
gram  managers,  and  weapon  system 
integration  engineers  communicate  sev¬ 
eral  times  a  week. 

•  46th  Operations  Group/Test  Squad¬ 
ron.  Responsible  Test  Organization  for 
mission  planning;  participates  in  our 
development  test  (DT),  DT/operational 
test  (OT),  and  formal  qualification  test 

(FQT). 

•  28th  Test  Squadron.  Final  test  audiori- 
ty  for  force  development  evaluation 
(FDE). 

•  Air  Force  Operational  Test  and  Eval¬ 
uation  Center  (AFOTEC)  Detach¬ 
ment  2.  Official  OT  agency  for  mission 
planning. 

•  AFOTEC  Detachment  5.  Official  OT 
agency  for  weapon  systems  like  the 
Cruise  Missiles  and  Joint  Air  to  Surface 
Stand-off  Missile  (JASSM). 

•  49th  Flight  Test  Squadron.  B-52 
Flight  Test  Squadron  at  Barksdale  AFB. 

•  5th  Operational  Support  Squadron. 
Warfighters  from  Minot  AFB. 

•  2nd  Operational  Support  Squadron. 
Warfighters  from  Barksdale  AFB. 

•  Mission  Planning  System  Support 
Facility.  Air  Force  mission  planning 
software  integration,  distribution,  and 
support  from  Hill  AFB. 

As  noted  above,  diroughout  die  applica- 
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Glossary 

A/W/E 

Aircraft/Weapons/Electronics 

FQT 

Formal  Qualification  Test 

ACM 

Advanced  Cruise  Missile 

ICD 

Interface  Control  Document 

AFMSS 

Air  Force  Mission  Support  System 

ICSMS 

Integrated  Conventional  Stores 

AFOTEC 

Air  Force  Operational  Test  and 

Management  System 

Evaluation  Center 

IDD 

Interface  Definition  Document 

AFPD 

Air  Force  Policy  Directive 

INTEL 

Intelligence  Data 

ACM 

Air-to-Ground  Missile 

JASSM 

Joint  Air-to-Surface  Standoff  Missile 

AISF 

Avionics  Integrated  Support  Facility 

JDAM 

Joint  Direct  Attack  Munitions 

ALCM 

Air  Launched  Cruise  Missile 

JSOW 

Joint  Standoff  Weapon 

AMI 

Avionics  Midlife  Improvement 

JTA 

Joint  Test  Assembly 

AWACS 

Airborne  Warning  and  Control  System 

OAS 

Offensive  Avionics  System 

CALCM 

Conventional  Air  Launched  Cruise  Missile 

OT 

Operational  Test 

CALCM  C/D 

Two  versions  of  the  CALCM 

PFPS 

Personal  Flight  Planning  System 

CLOAR 

Common  Low  Observability  Auto  Router 

SRAM 

Short  Range  Attack  Missile 

DAFIF 

Digital  Aeronautical  Flight  Information  File 

SEPG 

Software  Engineering  Process  Group 

DDLC 

Digital  Data  Loader  Cartridge 

SMXG 

Software  Maintenance  Group 

DT 

Development  Test 

SWPS 

Strategic  War  Planning  System 

DTC 

Data  Transfer  Cartridge 

TO 

Technical  Order 

DTUC 

Data  Transfer  Unit  Cartridge 

TBMCS 

Theater  Battle  Management  Core  System 

FDE 

Force  Development  Evaluation 

U2A 

U.S.  Strategic  Command  to  AFMSS 

FPM 

Flight  Performance  Model 

WCMD 

Wind  Corrected  Munitions  Dispenser 

tion’s  life  cycle  the  stakeholders  meet  fre- 
quendy  via  face-to-face  meetings  and  tele¬ 
conferences.  At  least  once  a  year,  a  mission 
planning  open  house  is  conducted  at  the 
user’s  base  of  operations.  This  includes  sev¬ 
eral  days  for  familiarization  with  our  existing 
applications  and  user  interface  working 
group  meetings  to  discuss  upcoming 
designs,  priorities,  trade-offs,  and  require¬ 
ments.  Our  engineers  and  computer  scien¬ 
tists  also  hit  the  road,  spending  an  average 
of  more  than  200  man-days  per  year  in  the 
field,  meeting  with  die  users,  oilier  develop¬ 


ers,  and  other  program  offices  to  continual¬ 
ly  coordinate  efforts,  schedules,  and  require¬ 
ments,  and  to  provide  familiarization  with 
our  systems. 

Requirements  review  boards  for  our 
A/W/E  applications  and  AFMSS  core  are 
conducted  and  defect  review  boards  are 
held  following  each  formal  test.  Eight  offi¬ 
cial  test  events  were  hosted  last  year.  Each 
event  had  one  or  more  users  or  customers 
working  side-by-side  with  the  developers.  As 
part  of  the  development  effort,  several  DTs 
are  hosted  prior  to  the  FQT.  At  least  one  of 


these  DT  tests  will  be  a  combined  DT / OT 
that  is  a  development  test  using  operational 
procedures,  data,  and  crew  members,  mak¬ 
ing  it  as  real-world  as  possible.  This  process 
is  a  profound  strength  in  the  organization. 
Feedback  is  received  from  the  users  contin¬ 
ually  throughout  the  development  phase. 

The  OT  certification  brief  is  prepared 
and  presented.  Using  Air  Force  Manual  63- 
119,  Certification  of  System  Readiness  for 
Dedicated  Operational  Test  and  Evaluation, 
a  matrix  of  33  certification  templates  is  eval¬ 
uated  that  identifies  specific  problem  or  risk 
areas  that  could  hinder  the  smooth  transi¬ 
tion  from  development,  through  test,  to  the 
fielding  of  a  product.  All  of  the  communi¬ 
ties  listed  above  participate  in  this  process, 
and  the  entire  group  agrees  that  the  product 
is  ready  to  be  tested.  This  final  step  provides 
complete  confidence  that  the  products  will 
meet  the  warfighter’s  expectations  the  first 
time,  every  time.  When  a  product  approach¬ 
es  fielding  certification,  the  actual  users  have 
seen  it,  used  it,  evaluated  it,  tested  it,  and 
stand  behind  it  along  with  the  developers. 

A  recent  success  story  centers  around  a 
flight  performance  change  to  the  way  mis¬ 
sion  planners  need  to  account  for  drag  on 
the  B-52  because  of  external  weapons  hang¬ 
ing  on  the  wings.  The  multiple  weapon  con¬ 
figurations  create  different  fuel  burn  rates 
affecting  the  range  of  the  aircraft. 
Implementing  this  change  in  the  TO 
spanned  three  software  releases  and 
required  participation  from  almost  every 
organization  listed  above. 

Through  requirements  review,  software 
development,  integration,  test,  and  the 


Figure  1 :  B-52  Mission  Planning  Environment 


M 

\FMSS  COF 
to  A/W/E  ICI 

21 

D  1 

► 

SJ 

LZ 

Smart  Weapon 
Planning  Modules 
(A/W/Es) 


A/W/E  TO 
A/W/E  I  CDs 

1  -  JDAM 

2  -  JASSM 

3  -AGM142 

4  -  WCMD 

5  -  JSOW 


1B52H-1 
1B52H-5 
1 B52H-1-12 
1 B52H-34-X 
1B52H-25-X... 


DDLC 

A _ 

DTC 

DTUC 

6  CROSSTALK  The  Journal  of  Defense  Software  Engineering 


August  2005 


Key  Elements  in  Fielding  Capabilities 


defect  review  board  process,  each  spiral 
would  furdier  refine  the  requirements,  each 
time  giving  die  user  increased  capabilities 
and  enhancing  the  system  effectiveness  to 
plan  aircraft  missions.  It  became  clear  early 
when  dealing  with  this  issue  that  each  mem¬ 
ber  of  the  team  had  a  different  piece  of  die 
puzzle.  Only  through  continuous  communi¬ 
cation  and  collaboration  were  the  answers  to 
all  the  questions  understood  enough  to  pro¬ 
duce  a  quality  tool  for  die  warfighter. 

From  start  to  finish,  nothing  is  done  in 
a  vacuum  widiout  the  user.  A  lot  of  com¬ 
panies  will  offer  the  user  an  early  look  or 
attendance  at  design  reviews,  but  the  cus¬ 
tomer  seldom  gets  die  complete  picture  or 
real  hands-on  experience  during  develop¬ 
ment  of  its  system.  As  explained  here, 
Tinker  AFB’s  B-52  mission  planning  sec¬ 
tion  goes  above  and  beyond  to  get  the  user 
involved  in  every  step.  Nothing  is  hidden 
or  kept  from  die  customer.  By  involving 
die  users  in  requirements  reviews,  early 
DT  events,  and  using  our  integrated  suite 
of  test  facilities,  die  customer  is  allowed  to 
actually  run  die  system  end-to-end  in  one 
location.  A  demonstration  of  one  or  two 
isolated  pieces  of  the  puzzle  is  not  needed 
because  die  user  sees  the  whole  enterprise 
and  walks  away  widi  a  feeling  of  confi¬ 
dence  that  what  is  paid  for  will  provide  die 
capabilities  required  in  the  field. 

This  type  of  team  effort  is  becoming 
more  important  with  the  Air  Force  Policy 
Directive  (AFPD)  63-1  cited  commander’s 
intent  diat  states,  “The  primary  mission  of 


our  acquisition  system  is  to  rapidly  deliver 
to  the  warfighter  affordable,  sustainable 
capability  diat  meets  dieir  expectations.” 
The  objective  of  AFPD  63-1  is  to  “create 
a  context  diat  allows  the  program  manager 
to  shape  and  execute  a  program  widi  an 
emphasis  on  teamwork,  trust,  common 
sense,  and  agility.”  It  further  states  that 
“the  warfighters,  developers/ acquirers, 
technologists,  testers,  budgeters,  sustainers, 
and  industry  must  plan  and  execute  togedi- 
er  in  order  to  meet  die  Commander’s 
intent.”  These  seem  to  be  lofty  ideals,  but 
we  have  proven  diey  can  be  done. 

Summary 

Fielding  capabilities  can  be  enhanced  by 
having  high  fidelity  test  environments  and 
by  collaboration  between  all  of  the  partic¬ 
ipants  on  a  program.  The  Air  Logistics 
Centers  at  Robins  AFB  and  Hill  AFB  have 
similar  capabilities  to  those  that  are 
described  in  this  article,  although  for  dif¬ 
ferent  weapon  systems.  Program  man¬ 
agers  may  be  able  to  take  advantage  of 
existing  organic  resources  to  reduce  cost 
and  risk.  Furdier,  partnering  agreements 
can  be  established  between  organic  soft¬ 
ware  support  activities  and  contractors  to 
facilitate  utilization  of  organic  resources 
in  teaming  arrangements  to  work  jointly 
on  Department  of  Defense  projects.  All 
diree  Air  Logistic  Center  software  sup¬ 
port  activities  have  Web  sites  (see  page  30) 
diat  provide  contacts.^ 


About  the  Authors 


John  D.  Holcomb  is 

the  Operational  Flight 
Programs  technical  ex¬ 
pert  in  the  76di  Soft¬ 
ware  Maintenance  Group 
at  the  Oklahoma  City- 
Air  Logistics  Center,  Tinker  Air  Force 
Base,  Okla.  He  has  40  years  of  federal 
service,  including  33  years  of  experi¬ 
ence  in  weapon  systems  software,  and 
participated  in  die  development  of  the 
Department  of  Defense  Standard 
2167.  He  has  a  bachelor’s  degree  in 
madi  and  physics  (double  major)  from 
the  University  of  Central  Oklahoma. 


76th  SMXG 
4750  Staff  DR 

Tinker  AFB,  OK  73145-3318 
Phone:  (405)  736-3835 
Fax:  (405)  736-3584 
E-mail:  john.holcomb@tinker.af.mil 


Group  at  Tinker  Air 
Force  Base,  Okla.  He  has  more  than  15 
years  experience  in  bodi  Test  Program 
Set  and  Operational  Flight  Program 
software  development  and  mainte¬ 


Michael  Hoehn  cur¬ 
rently  manages  die  B-52 
Mission  Planning  Soft¬ 
ware  Section  in  die  76th 
Software  Maintenance 


nance. 


76th  SMXG 

8080  Perimeter  RD  STE  107 

Tinker  AFB,  OK  73145 

Phone:  (405)  736-5539 

Fax:  (405)  736-4129 

E-mail:  michael.hoehn@tinker.af.mil 


Coming  Events 


September  12-16 

Practical  Software  Quality  and 
Testing  Conference  2005  North 
Minneapolis,  MN 

www.psqtconference.com/ 

2005north 

September  18-23 

International  Function  Point  Users 
Group  1 “  Annual  International  Software 
Measurement  and  Analysis  Conference 
New  Orleans,  LA 

www.ifpug.org/conferences/ 

annual.htm 

September  19-22 

Better  Software  Conference 
and  Expo  2005 
San  Francisco,  CA 

www.sqe.com/bettersoftwareconf 

September  26-27 

PDF  Conference  and  Expo 
Washington,  DC 

www.pdfconference.com 

October  16-20 

Object-Oriented  Programming,  Systems, 
Languages,  and  Applications  Conference 
San  Diego,  CA 

www.oopsla.org/2005/ShowPage. 
do?id  =  Home 

October  17-20 

MILCOM  2005 

Military  Communication  Conference 
Atlantic  City,  NJ 

www.milcom.org/2005 

October  17-21 

Quality  Assurance  Institute’s  26'’ 
Annual  Software  Testing  Conference 
Orlando,  FL 

www.qaiusa.com 

May  1-4,2006 

2006  Systems  and  Software 
Technology  Conference 

Uystems  &  Software 
Technology  Conference 

Salt  Lake  City,  UT 

www.stc-online.org 


August  2005 


www.stsc.hill.af.mil  7 


Delivering  Capabilities  Through  Partnerships 


Chris  D.  Moore 

Warner  Robins-Air  Logistics  Center 

As  public-private  partnerships  become  more  prevalent  in  the  Department  of  Defense  for  providing  logistics  support  for 
advanced  weapon  systems,  integrated  teams  must  look  for  unconventional  opportunities  to  exploit  the  best  capabilities  of  their 
combined  resources  to  support  many  diverse  program  objectives.  The  challenge  is  figuring  out  how  to  evolve  traditional  cus¬ 
tomer-supplier  relationships  into  truly  integrated  teams  with  common  objectives  at  the  forefront.  Warner  Robins-Air  Logistics 
Center  and  the  Northrop  Grumman  Corporation  are  pioneering  a  new  path  with  their  partnered  E-8C  Joint  STARS  soft¬ 
ware  maintenance  team. 


The  402d  Software  Maintenance  Group 
(SMXG,  formerly  MAS)  at  Warner 
Robins-Air  Logistics  Center  (WR-ALC), 
Robins  Air  Force  Base  (AFB),  Ga.,  is  soar¬ 
ing  high  in  its  partnership  with  Northrop 
Grumman  Systems  Corporation  (NGSC) 
to  maintain  the  complex  E-8C  Joint 
Surveillance  Target  Attack  Radar  System 
(Joint  STARS)  software  through  integrat¬ 
ed  teaming.  As  legislative  support  and 
expectation  for  government-private  part¬ 
nerships  continue  to  grow,  software 
organizations  are  finding  more  opportuni¬ 
ties  to  leverage  die  best  of  each  side’s 
infrastructure  to  provide  bodr  govern¬ 
ment  core  capability  and  optimum  sup¬ 
port  for  fielded  systems. 

The  challenge  is  figuring  out  how  to 
optimize  the  new  and  unconventional 
government-private  relationship.  While 
many  government-private  partnerships 
cast  the  parties  in  traditional  contractor- 
subcontractor  roles,  the  402  SMXG  and 
NGSC  partnership  has  taken  the  concept 
to  a  new  level  with  a  truly  integrated 
team  fielding  war-winning  capabilities 
through  software  maintenance. 

The  E-8C  Joint  STARS  Systems 
Group  (E8SG)  is  a  long-range  air-to- 
ground  surveillance  system  designed  to 
locate,  classify,  and  track  ground  targets  in 
all  weather  conditions.  It  is  operated  by 
die  1 1 6th  Air  Control  Wing  (ACW)  based 
at  Robins  AFB.  In  1998,  the  Air  Force 
designated  Joint  STARS  a  core  workload. 
The  Department  of  Defense  (DoD)  serv¬ 
ices  designate  certain  weapon  systems, 
equipment,  and  components  as  mission- 
essential  for  support  of  scenarios 
approved  by  the  Joint  Chiefs  of  Staff.  The 
DoD  ensures  diat  there  is  core  depot 
maintenance  capability  to  support  these 
mission-essential  weapon  systems.  Core 
exists  to  minimize  operational  risks  and  to 
guarantee  required  readiness  for  these 
weapon  systems. 

In  November  2000,  the  330  Intel¬ 
ligence  Reconaissance  and  Surveillance 
Group  (330  IRSG),  also  at  WR-ALC, 
awarded  a  Total  System  Support 


Responsibility  (TSSR)  contract  to  NGSC, 
which  was  the  Joint  STARS  system  inte¬ 
grator  operating  out  of  its  Airborne 
Ground  Surveillance  and  Battle 
Management  Systems  Division  in 
Melbourne,  Fla.  This  contract  made 
NGSC  responsible  for  maintaining  and 
providing  logistics  support  for  the  entire 
fleet  of  17  E-8C  jets  from  nose  to  tail.  In 
that  Joint  STARS  is  a  core  logistics  work¬ 
load,  a  partnership  was  required  between 
WR-ALC  and  NGSC  to  enable  WR-ALC 
to  support  NGSC’s  logistics  responsibili¬ 
ties  with  government- furnished  supplies 
and  services  (GFS/S).  While  this  part- 

“ Success  hinges  on  the 
deliberate  support 
of  all  objectives 
by  all  parties. ” 

nership  includes  the  support  of  many 
areas  of  logistics,  this  article  focuses  on 
Joint  STARS  software  maintenance. 

The  fiscal  year  1998  Defense 
Authorization  Act  provided  statutory 
authority  for  DoD  depots  to  enter  into 
government-private  cooperative  arrange¬ 
ments  (partnerships)  for  the  perform¬ 
ance  of  depot-level  work.  While  the  con¬ 
cept  of  partnering  or  teaming  with 
industry  was  conceived  initially  as  a  strat¬ 
egy  for  depots  to  make  skills,  facilities, 
and  equipment  available  to  the  private 
sector  to  perform  government  workload 
and  to  maximize  the  utilization  of  skills, 
facilities,  and  equipment  that  are  required 
to  support  core  workloads,  it  lent  itself 
well  to  a  somewhat  different  arrange¬ 
ment  regarding  Joint  STARS  software 
maintenance.  In  this  case  the  contractor, 
NGSC,  had  a  history  of  maintaining 
Joint  STARS  software  almost  exclusively. 
Further,  the  facilities  and  expertise  exist¬ 
ed  at  the  contractor  facility  in 


Melbourne,  not  at  Robins  AFB.  In  this 
case,  the  government  is  the  novice  and 
the  contractor  is  the  expert. 

As  an  umbrella  to  the  WR-ALC- 
NGSC  partnership,  a  Long  Range 
Memorandum  of  Agreement  (LRMOA) 
was  established  between  the  three  princi¬ 
ple  parties  depicted  in  Figure  1 .  They  are 
WR-ALC,  specifically  the  402d 
Maintenance  Wing  (MXW),  formerly  WR- 
ALC/Maintenance  Directorate;  the  E8 
Systems  Group,  formerly  tire  Joint  STARS 
System  Program  Office  at  Electronics 
System  Command  (ESC);  and  NGSC. 

The  agreement  set  die  ground  rules 
for  partnering  by  acknowledging  the  exis¬ 
tence  of  bodr  common  and  unique  indi¬ 
vidual  objectives.  While  there  are  mutual 
objectives  in  government-private  partner¬ 
ships,  there  are  many  objectives  that  are 
unique  to  the  parties  and  may  have  higher 
priorities  to  the  respective  parties. 

Success  hinges  on  the  deliberate  sup¬ 
port  of  all  objectives  by  all  parties.  The 
challenge  is  pioneering  innovative  medi- 
ods  for  achieving  these  objectives  that 
sometimes  appear  to  be  in  complete  con¬ 
flict.  The  agreement  recognized  the  new 
and  unique  roles  of  the  parties  as  well  as 
individual  objectives,  but  most  important¬ 
ly  set  forth  the  agreement  to  pursue  a  set 
of  common  objectives. 

The  system  program  director  E8SG 
objectives  are  to  implement  U.S.  Air  Force 
(USAF)  program  management  and  acqui¬ 
sition  techniques  as  necessary  to  provide 
the  best  possible  support  to  the  warfight¬ 
er  while  ensuring  that  best  value  is 
achieved  for  the  USAF.  The  focus  of 
E8SG  is  on  providing  die  user  with  supe¬ 
rior  combat  capability  by  balancing  pro¬ 
gram  costs,  schedule,  and  performance 
elements. 

The  WR-ALC  objectives  are,  pursuant 
to  law,  to  maintain  the  appropriate  levels 
of  core  competencies  in  logistics  manage¬ 
ment,  engineering,  supply  chain  manage¬ 
ment,  and  depot-level  maintenance  neces¬ 
sary  to  ensure  efficient  and  cost-effective 
sustainment  for  all  of  its  assigned  weapon 
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systems  now  and  in  the  future.  To  achieve 
these  objectives,  the  center  must  develop 
in-house  expertise  and  transition  software 
maintenance  workload  from  the  contrac¬ 
tor  to  organic  while  mitigating  risk  to  the 
user  and  to  the  overall  program.  Success 
in  this  endeavor  satisfies  the  core  directive. 

An  additional  WR-ALC  objective  is  to 
partner  with  the  contractor  to  reduce  the 
core  capital  investment  through  sharing  of 
equipment  and  resources. 

Contracting  Objectives 

As  the  TSSR  contractor,  NGSC  objec¬ 
tives  are  to  fulfill  an  important  role  in 
assisting  the  USAF  organizations  in 
achieving  their  respective  objectives. 
Consequently,  NGSC’s  primary  objective 
is  to  ensure  the  Joint  STARS  weapon  sys¬ 
tem  continues  to  provide  the  user  with 
superior  and  reliable  combat  capability. 
This  performance,  in  turn,  allows  NGSC 
to  perform  its  TSSR  contract  obligations. 
By  providing  superior  Joint  STARS  sup¬ 
port,  NGSC  establishes  a  positive  reputa¬ 
tion  for  the  Joint  STARS  weapon  system, 
earns  maximum  performance  incentives, 
and  enhances  its  reputation  as  a  leading 
provider  of  Joint  STARS  support. 

When  these  individual  objectives  are 
superimposed,  a  set  of  common  objec¬ 
tives  comes  into  focus.  The  parties’  intent 
is  that  through  this  LRMOA,  and  other 
subordinate  agreements,  the  parties  will 
successfully  integrate  their  efforts  to  con¬ 
tinuously  improve  total  systems  support 
of  Joint  STARS.  Notwithstanding  the 
unique  roles  and  responsibilities  of  each 
independent  party,  the  parties  collectively 
agree  that  their  overarching  mutual  objec¬ 
tives  are  to  provide  the  following: 

•  Superior  support  to  the  warfighter. 

•  Best  value  to  the  USAF  (balancing 
both  program  and  broader  objec¬ 
tives). 

•  The  mechanisms  necessary  to  meet 
the  USAF  core  logistics  competencies 
requirements. 

•  Support  to  future  core  and  source-of- 
repair  assignment  analyses  and  deci¬ 
sion-making  to  enable  the  USAF  to 
balance  objectives  of  the  Joint 
STARS  program  with  broader  USAF 
imperatives  such  as  the  maintenance 
of  core  competencies. 

•  The  creation  of  an  integrated  digital 
environment  (IDE)  as  a  key  enabler 
in  achieving  the  communication, 
coordination,  insight,  and  responsive¬ 
ness  objectives  outlined  in  this  docu¬ 
ment  (discussed  later). 

•  The  ability  of  NGSC  to  achieve  rea¬ 
sonable  profits  and  enhance  its  cor¬ 
porate  reputation  through  demon- 


Software  Transition  Plan 

Figure  1 :  Long  Range  Memorandum  of  Agreement 


strated  performance  in  achieving  the 
aforementioned  common  objectives 
as  applicable  to  the  TSSR  contract. 
Clearly  there  are  many  objectives  ripe 
for  competing  interest.  The  TSSR  con¬ 
tractor  must  balance  maximizing  corpo¬ 
rate  performance  with  supporting  the  Air 
Force  in  meeting  its  core  capability 
requirements.  The  government  partner 
must  acquire  laboratories,  personnel,  and 
training,  and  develop  expertise  and  tran¬ 
sition  workload  while  mitigating  program 
risks.  It  is  a  tough  challenge,  but  major 
risk  mitigators  include  early  identification 
and  optimization  of  individual  strengths, 
practical  integration  of  engineering  and 
management  processes  and  human 
resources,  implementation  and  continued 
optimization  of  communication  process¬ 
es  and  procedures,  and  identification  of 
weak  areas  followed  by  development  of 
tactical  plans  to  effectively  fill  those 
voids.  Intense  focus  in  these  areas  with 
little  regard  for  organizational  bound¬ 
aries  and  contractor/government  affilia¬ 
tions  produces  not  only  a  partnered 
team,  but  also  an  integrated  team  pos¬ 
tured  to  improve  efficiency,  perform¬ 
ance,  and  quality  —  a  win-win  scenario. 
The  bottom  line:  all  objectives  are  met 
and,  most  importantly,  superior  warfight¬ 
er  support  is  provided. 

There  is  also  the  concept  of  trust. 
Traditionally  the  government  personnel 
have  been  perceived  as  the  customer  in  all 
instances.  The  partnership  scenario  casts 
the  government  partner  in  an  entirely  dif¬ 
ferent  role:  an  insider,  a  colleague,  and  yes, 
in  some  forms  even  as  a  subordinate.  Not 
only  is  this  scenario  foreign  to  the  con¬ 
tractor,  but  extremely  new  and  possibly 
uncomfortable  for  the  government  per¬ 
sonnel.  It  takes  deliberate  attention  and 


actions  to  bridge  this  gap. 

The  contractor  developed  the  E-8C 
and  remains  fiercely  loyal  to  it,  coveting 
it  and  its  caretaking.  In  many  cases,  it  is 
the  lifeblood  of  the  company  or  depart¬ 
ment  and,  to  some  extent,  its  surround¬ 
ing  community.  Eradicating  the  culture 
of  competition  is  essential  to  the  success 
of  the  government-private  partnership, 
but  this  cannot  be  mandated  or  accom¬ 
plished  externally.  It  can  only  come  from 
within  the  integrated  team  through 
building  the  spirit  of  teamwork  and 
shared  goals  as  reported  in  the  following 
section. 

Working  Together 

While  many  obstacles  can  be  anticipated 
at  the  outset  and  contingencies  devised, 
most  lie  in  waiting  just  around  the  corner 
and  can  initially  appear  to  be  insur¬ 
mountable.  To  name  a  few,  there  is  geo¬ 
graphical  separation,  differences  in  soft¬ 
ware  laboratory  setup  and  capability,  dif¬ 
ferent  technical  and  management 
processes,  risk  to  contractor  perform¬ 
ance  due  to  the  government’s  poor  per¬ 
formance,  competition  versus  teamwork, 
and  the  previously  discussed  core  objec¬ 
tives  versus  best  value. 

The  E-8C  is  essentially  a  spiral  devel¬ 
opment  system  with  continuous  develop¬ 
ment  of  new  system-level  enhancements 
and  functionality,  and  subsequent  field¬ 
ing  of  these  new  capabilities  through 
integration  into  the  software  mainte¬ 
nance  process.  Concurrent  with  new 
development  is  ongoing  identification  of 
software  change  requirements  to  address 
deficiencies  and  defects,  which  is  the  cus¬ 
tomary  software  sustainment  life  cycle. 

The  first  four  years  of  the  partner¬ 
ship  took  a  natural  course  with  respect  to 
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software  workload  sharing  and  transi¬ 
tion.  The  402  SMXG  team  had  experi¬ 
ence  in  performing  limited  modifications 
to  the  Joint  STARS  software  specifically 
in  the  area  of  ground  support  software 
tools.  These  tools  varied  from  pre-  and 
post-mission  support  tools  to  software 
development  and  test  tools.  At  first, 
NGSC  tasked  the  402  SMXG  to  perform 
software  change  projects  in  these  areas. 
As  the  system  integrator  prior  to  the 
partnership,  NGSC  integrated  -  for  a  fee 
—  any  organically  produced  software 
modifications,  thereby  limiting  organic 
contributions  to  the  software  mainte¬ 
nance  effort. 

Under  the  TSSR  contract  and 
through  the  partnership,  NGSC  engi¬ 
neering  management  opened  the  doors 
to  the  NGSC  software  lab  and  invited  the 
Robins  software  engineers  to  deliver  and 
integrate  their  own  software  changes. 
This  single  activity  laid  the  foundation 
for  what  would  become  the  standard 
software  integration  process.  The  402 
SMXG  engineers  would  become  respon¬ 
sible  for  following  NGSC  software  engi¬ 
neering  processes  for  in-house  software 
delivery,  integration,  and  testing.  It  also 
was  the  first  step  toward  building  confi¬ 
dence  by  the  TSSR  contractor  in  the  gov¬ 
ernment  software  maintenance  team. 

Prior  to  the  partnership  and  the  TSSR 
contract,  software  requirements  were 
flowed  to  NGSC  in  a  very  structured  and 
formal  process  of  requirements  manage¬ 
ment  by  the  operational  user,  the  116th 
ACW  and  the  330  IRSG.  Under  the  TSSR 
contract,  NGSC  took  on  a  more  program¬ 
matic  role  and  the  responsibility  for  mak¬ 
ing  capability-  and  value-based  decisions 
with  regard  to  requirements  management. 
NGSC  established  several  requirements 
working  groups  in  specific  functional  areas 
of  the  software.  These  working  groups 
consisted  of  representatives  from  all  stake¬ 
holders,  including  the  user,  contractor, 
government  team,  and  program  office. 


The  working  groups  are  responsible 
for  evaluating  software  problems,  alter¬ 
native  solutions,  and  ultimately  recom¬ 
mending  requirements.  Involvement  by 
the  government  software  team  in  these 
forums  is  essential  to  establishing  a  core 
software  maintenance  capability  and  has 
proven  invaluable  to  the  government 
team.  Final  requirements  are  managed  in 
Integrated  Release  System  Engineering 
Management  Teams  (IR  SEMT).  While 
this  forum  is  the  official  communication 
between  the  contractor  and  the  program 
office  for  software  requirements,  the  402 
SMXG  is  a  participant  as  well. 

Probably  as  significant  as  anything 
has  been  the  inclusion  of  the  402  SMXG 
in  day-to-day  software  maintenance  team 
meetings.  These  vary  in  function,  but  are 
numerous,  and  all  are  critical  to  continu¬ 
ous  and  optimum  performance  by  the 
integrated  software  maintenance  team. 
There  are  weekly  Administrative  SEMT 
meetings  where  project  status  and  risks 
are  discussed  and  short-term  direction  is 
provided.  The  weekly  Technical  SEMT  is 
chaired  by  the  Joint  STARS  software 
chief  architect.  In  these  forums,  engi¬ 
neering  leads  and  subject-matter  experts 
discuss  current  tasks  and  issues  from  a 
system  perspective,  and  technical  guid¬ 
ance  is  cultivated  and  provided.  In  the 
weekly  software  integrated  product  team 
(IPT)  meetings,  software  change  designs 
are  reviewed  by  leads  from  each  system 
area  to  determine  correctness  prior  to 
approval  at  the  weekly  software  configu¬ 
ration  control  board  (CCB).  In  the  CCB, 
completed  work  is  approved  for  incorpo¬ 
ration  into  the  weekly  software  build,  and 
new  assignments  are  made  based  on 
mature  requirements  and  software 
change  requests. 

As  you  can  see,  the  integrated  team’s 
concept  of  operation  for  a  particular  soft¬ 
ware  release  is  one  of  continuous  problem 
analysis,  software  modification,  integra¬ 
tion,  and  test.  Major  requirements  that 


flow  from  the  IR  SEMT  are  developed 
and  integrated  simultaneously  with  the 
continuous  correction  of  defects.  Major 
requirements  dictate  the  overall  integrated 
release  schedule  and  determine  when  sig¬ 
nificant  modification  curtails. 

While  NGSC  has  primary  responsibil¬ 
ity  for  conducting  system-level  testing  on 
the  E-8C,  the  402  SMXG  participates  in 
this  area  as  well  to  further  the  develop¬ 
ment  of  government  expertise  and  capa¬ 
bility.  In  this  phase,  NGSC  must  balance 
mentoring  with  meeting  system  require¬ 
ments  and  schedule.  The  402  SMXG  has 
performed  various  roles  ranging  from 
mission  planning  using  the  ground  soft¬ 
ware  tools  to  actually  flying  on  test  aircraft 
and  performing  system-level  tests. 

The  402  SMXG  possesses  an  extraor¬ 
dinary  benefit  in  being  co-located  with  the 
single  Joint  STARS  operational  wing,  the 
1 16th  ACW  Not  only  does  the  402  SMXG 
reside  in  the  same  building  as  the  116th 
Computer  Systems  Squadron  that  per¬ 
forms  pre-  and  post-mission  operations  as 
well  as  software  requirements  manage¬ 
ment,  but  the  402  SMXG  also  operates 
and  maintains  the  Joint  STARS  mission 
crew  training  systems  in  the  same  facility. 
Access  to  this  critical  resource  enables  the 
402  SMXG  to  test  the  IR’s  weekly  devel¬ 
opmental  builds  on  a  target  system.  This 
can  be  done  because  Joint  STARS  incor¬ 
porates  the  concept  of  a  single  software 
baseline.  The  same  software  drat  executes 
on  the  E-8C  also  executes  on  the  trainers. 
Further,  the  402  SMXG  has  the  responsi¬ 
bility  of  ensuring  the  IR  will  operate 
according  to  specification  on  the  training 
system  once  complete. 

Achieving  Full  Operational 
Capability 

As  stated  before,  the  402  SMXG  has 
benefited  immensely  by  partnering  with 
the  Joint  STARS  integrator  and  being 
allowed  to  perform  as  an  integrated 
teammate  rather  than  as  a  subcontractor. 
This  natural  teaming  process  has  enabled 
processes  and  relationships  to  form  from 
the  ground  up  rather  than  be  dictated 
from  above.  However,  in  the  fourth  year 
it  became  obvious  that  a  quantitative 
structure  was  needed  to  evaluate  when 
the  402  SMXG  would  be  certified  as 
capable  of  performing  the  software 
maintenance  workload;  in  other  words, 
achieving  full  operational  capability 
(FOC).  A  Core  Activation  Plan  (CAP) 
was  established  to  provide  the  FOC  plan 
and  criteria.  The  FOC  methodology  con¬ 
sists  of  a  scoring  mechanism  to  evaluate 
the  government  software  maintenance 
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Robins  Team  -  Warner  Robins  Air  Force  Base,  Georgia 


Unclassified  Information 

•  Restricted  access  Web  site  (Secure  Socket  Layers) 

•  Livelink  Intranet 

Classified  Information 

•  Secure  Link  (Leased  T-1  with  Encryption) 

•  Common  Configuration  Management  System 


NGC  Team  -  Melbourne,  Florida 


NGC  manages  configuration  control  for  both  sites 
from  Melbourne;  ESC/JS  CCB  approves  releases. 


Figure  3:  The  Robins  AFB  and  NGC  Team  Initial  IDE 


capability  against  five  critically  weighted 
areas  as  depicted  in  Figure  2.  Those  areas 
include  1)  core  workload,  2)  technical 
expertise,  3)  software  laboratory,  4)  man¬ 
agement,  and  5)  processes. 

This  status  against  this  CAP  is  pre¬ 
sented  to  WR-ALC,  NGSC,  SPO,  and  the 
116th  ACW  senior  leadership  semiannual¬ 
ly  in  the  Joint  STARS  Partnership  Review 
Committee  briefing.  Upon  performing  the 
initial  FOC  evaluation,  we  learned  that  the 
area  needing  the  most  attention  was  tech¬ 
nical  expertise.  Mentoring  would  be 
required  from  the  contractor  to  the  gov¬ 
ernment  team  in  specific  areas  that  are  sta¬ 
ble  with  minimal  modification,  and  in  spe¬ 
cific  areas  where  the  government  team 
lacked  sufficient  experience  and  had  per¬ 
formed  minimal  or  no  software  changes. 

To  complete  the  picture,  deliberate 
steps  were  required.  NGSC  increased  the 
level  of  software  maintenance  workload 
assigned  to  the  402  SMXG  in  these  areas 
and  increased  the  level  and  quality  of 
mentoring  of  government  engineers. 
This  initiative  ensures  quality  software  is 
delivered  on  time  in  addition  to  transfer¬ 
ring  much  needed  knowledge  to  the  402 
SMXG  —  again  a  win-win  approach 
through  partnering. 

A  final,  and  critically  important  com¬ 
mon  objective,  was  the  establishment  of 
an  IDE.  This  goal  was  conceived  at  part¬ 
nership  inception,  but  has  been  the  most 
illusive.  The  IDE,  as  depicted  in  Figure  3, 
was  planned  to  be  an  electronic  collabora¬ 
tive  workspace  to  facilitate  joint  and 
simultaneous  accomplishment  of  software 
engineering  at  geographically  separated 
sites  manned  by  separate  teams.  For  it  to 
become  a  reality,  the  IDE  required  soft¬ 
ware  equipment  laboratory  upgrades  on 
both  ends,  security  approvals,  integration 
and  testing,  and  concepts  of  operation. 

During  the  first  four  years  of  the 
partnership,  we  relied  upon  weekly 
overnight  shipments  between  Robins 
AFB  and  Melbourne  to  facilitate  the 
delivery  of  software  products  to  and 
from  the  two  sites.  In  January  2005,  our 
IDE  was  completed  with  official  opera¬ 
tion  of  a  network  connection  between 
the  two  sites.  We  will  use  the  connection 
to  transfer  software  files  back  and  forth 
to  facilitate  integrated  software  mainte¬ 
nance  teaming,  and  also  to  connect 
remotely  to  each  site  for  day-to-day  oper¬ 
ations.  We  are  one  step  closer  to  becom¬ 
ing  a  virtual  integrated  team  via  the  com¬ 
pletion  of  this  connection. 

Conclusion 

Having  said  this,  let  us  take  a  look  at  our 
current  status.  We  have  integrated 


processes  and  project  management,  we 
have  devised  a  method  of  sharing  soft¬ 
ware  workload,  we  have  made  plans  to 
shore  up  the  government’s  weak  areas  in 
system  expertise,  and  we  have  completed 
the  initial  IDE.  The  402  SMXG’s  current 
score  against  the  FOC  criteria  estab¬ 
lished  in  the  CAP  is  95  percent  out  of  a 
required  90  percent.  Upon  completion  of 
the  software  release  currently  in  develop¬ 
ment,  the  partners  plan  to  declare  the 
government  capability  fully  operational. 

Where  do  we  go  from  here?  While 
the  initial  period  of  the  partnership  has 
focused  heavily  on  establishing  organic 
software  maintenance  capability  concur¬ 
rently  with  satisfying  user  requirements, 
the  future  likely  holds  in  store  shrinking 
budgets  and  uncertainty.  The  partnered 
team  will  need  to  increase  attention  on 
value  and  optimization  on  both  ends. 
With  the  next  generation  Joint  STARS  — 
the  E-10  —  on  the  distant  horizon,  the 
current  platform  will  likely  continue  to 
undergo  upgrades  with  the  addition  of 
new  functionality  to  maintain  pace  with 
the  demands  of  global  conflict. 

NGSC  will  be  called  upon  to  develop 
new  capabilities  that  exploit  the  system 
for  maximum  operational  effectiveness. 
The  402  SMXG  will  focus  on  increasing 
knowledge  base  and  efficiency  to  keep  up 
with  the  flow  of  software  maintenance 
requirements.  Both  teams  will  look  for 
opportunities  to  reduce  costs  and  opti¬ 
mize  processes.  Exploiting  the  IDE  will 
be  a  central  focus  for  the  near  term.  Joint 
efforts  to  strengthen  and  optimize  inter¬ 
faces  with  the  user  to  improve  software 
maintenance  requirements  management 
will  also  be  a  central  focus  of  the  part¬ 
nership. 

®  Capability  Maturity  Model  is  registered  in  the  U.S.  Patent 
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Sharing  and  applying  lessons  learned 
to  other  partnered  programs  within  the 
Air  Force  Materiel  Command  will  be  a 
significant  goal  for  the  coming  years. 
While  the  future  was  initially  somewhat 
blurry,  the  Joint  STARS  software  mainte¬ 
nance  partners  have  exploited  their  new¬ 
found  relationship  to  bring  world-class 
war  fighting  capability  and  value  for  the 
warfighter  into  focus. ♦ 
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The  Department  of  Defense’s  increasing  dependence  on  new  technology  such  as  unmanned  aerial  vehicles  has  created  a  need  for 
high-assurance  systems  that  deliver  the  strongest  degree  of  security  and  safety.  However,  there  has  been  a  notable  lack  of  sys¬ 
tems,  commercial  or  government-sponsored  research,  and  engineering  that  meet  these  requirements.  The  Multiple  Independent 
Levels  of  Security  (MILS)  architecture  is  proposed  as  a  solution  to  meet  the  needs  for  critical  information  assurance.  MILS 
is  a  componentiged  architecture  based  on  a  conimercial  off-the-shelf  separation  kernel  that  enforces  strict  communication  and 
partitioned  process  execution.  MILS  supports  multiple  levels  of  security  communication,  security  policy  composition,  and  mod¬ 
ular  design  so  that  critical  components  are  able  to  be  evaluated  at  the  highest  levels  to  ensure  secure  and  safe  operation. 


Information  supremacy  wins  wars. 

Warfare  has  always  required  sharing  the 
right  information  with  die  right  person  at 
die  right  time.  Technology  today  enables 
information  sharing  on  a  scale  well  beyond 
what  our  forefathers  imagined,  but  sharing 
information  with  die  wrong  individuals  can 
have  catastrophic  consequences. 

Secure  information  sharing  is  critical  to 
enable  and  protect  the  warfighter  widiout 
compromising  die  mission.  The  challenge 
is  diat  warfighter-crucial  information  is 
highly  diverse.  Initiatives  such  as  Network- 
Centric  Warfare,  System  of  Systems,  and 
die  Global  Information  Grid  strengdien 
die  desire  to  share  information  with  multi¬ 
ple  levels  of  security  (MLS).  MLS  systems 
have  historically  been  among  die  most  chal¬ 
lenging  and  expensive  systems  to  develop 
and  deploy  [1].  Sharing  and  separating 
information  in  coalition  force  operations  is 
an  equally  challenging  and  furdier  compli¬ 
cating  problem. 

Multiple  Independent  Levels  of  Security 
(MILS)  is  an  architecture  that  makes  devel¬ 
opment,  accreditation,  and  deployment  of 
MLS-capable  systems  more  practical, 
achievable,  and  affordable.  The  MILS 
architecture  significantly  increases  protec¬ 
tion,  reduces  time  to  develop,  and  reduces 
schedule  risk  of  deploying  technology  to 
provide  high-assurance  systems  diat  are 
both  safe  and  secure  [1], 

While  die  MILS  architecture  allows  for 
a  system  of  highly  secure  distributed  com¬ 
ponents,  it  does  not  automatically  guaran¬ 
tee  a  secure  composed  system  out  of  inde¬ 
pendent  secure  components.  System-wide 
security  is  still  up  to  die  system  designer 
widi  MILS  providing  die  building  blocks 
and  tools  needed  to  construct  a  system- 
level  security  policy  diat  can  then  be  veri¬ 


fied.  There  are  tools,  e.g.,  Boundary  Flow 
Modeling  [2],  for  assuring  diat  security  poli¬ 
cies  compose  for  a  given  system. 

Where  We  Have  Been 

In  almost  all  commercial  off-the-shelf 
(COTS)  operating  systems  and  communica¬ 
tions  technologies,  security  is  an  after- 
diought,  addressed  via  a  fail-first,  patch-later 
paradigm.  When  a  system  is  penetrated, 
fixes  are  then  pursued  to  plug  the  hole. 
After  a  new  virus  propagates,  die  enabling 
weakness  is  repaired  to  stop  furdier  infec¬ 
tion.  The  frail  approach  of  attempting 
repair  of  infected  systems  is  also  common. 
Damage  is  frequendy  not  detectable  or 
repairable  in  systems  with  weak  security 
foundations.  Fail-first,  patch-later  is  inap¬ 
propriate  for  any  mission-critical  system 
because  it  is  reactive  and  always  one  step 
behind  the  attacker.  In  mission-critical  sys¬ 
tems,  damage  must  be  avoided  or  bounded 
when  impossible  to  avoid.  Proactive  meas¬ 
ures  are  required  to  safeguard  information 
and  the  warfighter,  and  prevent  the  damage 
from  happening  in  the  first  place. 

In  traditional  architectures,  diere  were 
good  reasons  to  assign  all  policy  enforce¬ 
ment  to  a  monolidiic  security  kernel.  To 
ensure  diat  enforcement  was  non-bypass- 
able,  security  functions  had  to  be  part  of 
every  system  service  request.  To  ensure  diat 
enforcement  was  tamper-proof,  security 
functions  had  to  be  in  an  address  space  sep¬ 
arate  from  die  application  [3],  These  securi¬ 
ty  functions  needed  to  be  in  a  monolidiic 
security  kernel  since  die  computing  power 
of  two  decades  ago  was  not  sufficient  to 
perform  die  context  switches  required  to 
separate  all  of  diese  processes  and  data  and 
still  maintain  system  performance. 

The  security  kernel  widi  its  set  of  trust¬ 


ed  security  functions  often  produced  large, 
complex,  unstructured  programs  diat  were 
difficult  to  certify  at  die  higher  assurance 
levels  [4].  SCOMP,  which  managed  to 
achieve  die  highest  A1  security  rating  via 
the  historic  “Orange  Book”  [5]1,  was  based 
on  a  very  simple  security  kernel  [3].  Most 
security  kernel-based  systems  never 
achieved  die  highest  level  of  certification, 
which  required  formal  verification.  The 
Motorola  Network  Encryption  System  that 
handled  MLS  data  through  encryption 
achieved  a  much  lower  B2  rating,  and  the 
XTS-300,  die  successor  to  SCOMP,  was 
certified  at  a  B3  rating  [3], 

Aside  from  die  difficulty  of  certifying 
systems  widi  complex,  monolidiic  kernels, 
the  more  important  problem  is  in  trying  to 
enforce  a  single,  system-wide  security  poli¬ 
cy.  For  example,  Blacker,  which  successful¬ 
ly  handled  encryption  of  MLS  data,  could 
not  successfully  accommodate  administra¬ 
tive  traffic  widiin  its  model  of  classification 
levels  [3],  In  general,  security  policies  in  the 
kernel  did  not  provide  the  robustness 
required  for  many  applications  where  appli¬ 
cation-specific  security  policies  would  have 
provided  more  tighdy  focused  protection. 

Where  We  Are  Going 

MILS  was  created  to  enable  application- 
level  security  engineering  at  a  high  level  of 
assurance  while  being  affordable.  MILS 
takes  advantage  of  Moore’s  Law’s  [6]  per¬ 
formance  increases  over  the  last  two 
decades  by  layering  small,  formally  mod¬ 
eled  and  mathematically  verified  com¬ 
ponents  togedier  to  create  a  high-assurance 
foundation.  In  MILS,  applications  are 
empowered  to  enforce  their  own  security 
policies  instead  of  relying  on  generalized 
kernel  security  services.  MILS  also  enables 
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efficient  systems  engineering,  where  high- 
assurance  components  can  be  effectively 
reused  without  modification.  This  lowers 
certification  costs  since  certification  arti¬ 
facts  can  be  reused. 

The  concept  of  MILS  originated  in 
papers  written  by  John  Rushby,  Ph.D,  of 
the  Stanford  Research  Institute  in  the  early 
1980s  [7,  8].  Rushby  proposed  that  a  sepa¬ 
ration  kernel  divide  memory  into  partitions 
using  the  hardware  memory  management 
unit  and  allow  only  carefully  controlled 
communications  between  non-kernel  parti¬ 
tions.  This  allows  one  partition  to  provide  a 
service  to  another  with  minimal  interven¬ 
tion  from  the  kernel  [7]. 

Traditional  operating  system  services 
drat  previously  ran  in  privileged  (i.e.,  super¬ 
visor)  mode  such  as  device  drivers,  file  sys¬ 
tems,  network  stacks,  etc.,  now  run  in  non- 
privileged  (i.e.,  user)  mode.  Because  a  sepa¬ 
ration  kernel  provides  very  specific  func¬ 
tionality,  the  security  policies  that  must  be 
enforced  at  this  level  are  relatively  simple. 
The  primary  concerns  of  a  separation  ker¬ 
nel  are  the  partitioning  of  processes  and 
data  plus  the  containment  of  systemwide 
failures.  Consequently,  we  can  capture  the 
security  requirements  for  a  separation  ker¬ 
nel  by  four  foundational  security  policies: 

•  Data  Isolation.  Information  in  a  parti¬ 
tion  is  accessible  only  by  that  partition, 
and  private  data  remains  private. 

•  Control  of  Information  Flow.  Infor¬ 
mation  flow  from  one  partition  to 
another  is  from  an  authenticated  source 
to  authenticated  recipients;  the  source 
of  information  is  authenticated  to  the 
recipient,  and  information  goes  only 
where  intended. 

•  Periods  Processing.  The  microproces¬ 
sor  and  any  networking  equipment  can¬ 
not  be  used  as  a  covert  channel  to  leak 
information  to  listening  third  parties. 

•  Fault  Isolation.  Damage  is  limited  by 
preventing  a  failure  in  one  partition 
from  cascading  to  any  other  partition. 
Failures  are  detected,  contained,  and 
recovered  locally. 

The  resultant  kernel  is  now  much  small¬ 
er  and  simpler,  and  conducive  to  rigorous 
inspection  and  mathematical  proof  of  cor¬ 
rectness  by  techniques  such  as  formal 
methods.  This  size  reduction  is  an  instanti¬ 
ation  of  MILS’s  most  fundamental  benefit: 
Fracnaticalty  reduce  the  amount  of  security-critical 
code  so  that  we  can  dra?naticalty  increase  the  level  of 
rigor  when  we  inspect  that  code.  If  we  are  doing 
very  few  things,  we  should  be  able  to  do 
them  very  well,  so  well,  in  fact,  that  the 
code  can  be  trusted  to  protect  our  most  valu¬ 
able  data  under  die  highest  level  of  threat. 

MILS  middleware  is  an  expansive  con¬ 
cept  with  a  very  broad  user-mode  layer.  It 


contains  many  operating  system  services 
such  as  device  drivers  that  previously  ran  in 
privileged  mode.  Running  in  user  mode, 
drey  are  subject  to  the  kernel’s  security  pol¬ 
icy  enforcement.  MILS  middleware  also 
includes  functions  traditionally  thought  of 
as  being  one  level  removed  from  the  core 
operating  system:  file  systems,  network 
stacks,  common  libraries,  encryption, 
authentication,  etc.  MILS  middleware  also 
includes  traditional  application-level  mid¬ 
dleware  technologies  such  as  Common 
Object  Request  Broker  Architecture 
(CORBA)  [8],  Data  Distribution  Service 
(DDS),  Web  services,  etc.  MILS  middle¬ 
ware  resides  in  the  same  user-mode  parti¬ 
tion  as  the  application  that  it  supports  or  in 
protected  user-mode  partitions  by  itself. 

Applications  do  their  processing  and 
enforce  their  own  security  policies  in  user-mode 
partitions.  Applications  running  in  their 
partitions  can  only  access  the  memory  that 
has  been  explicitly  allocated  for  each  parti¬ 
tion.  Application  partitions  can  only  com¬ 
municate  with  each  other  through  paths 
that  have  been  configured  when  the  sys¬ 
tem  was  generated.  Under  no  circum¬ 
stances  may  application  partitions  access 
hardware  directly  unless  explicitly  author¬ 
ized  to  do  so.  The  MILS  architecture,  along 
with  a  notional  set  of  allowable  informa¬ 
tion  flows,  is  illustrated  in  Figure  1 . 

Why  is  application-level  security-policy 
enforcement  effective  in  MILS  when  it  was 
not  effective  by  itself  in  traditional  mono¬ 
lithic  architectures?  It  is  because  the  MILS 
separation  kernel  guarantees  control  of 
information  flow  and  data  isolation.  It 
makes  this  guarantee  for  the  first  time  at  an 
assurance  level  that  was  next  to  impossible 
to  achieve  with  the  monolithic  kernels.  Due 
to  technology  advances  in  both  smaller  cir¬ 
cuits  and  increased  functionality,  we  now 
have  processors  powerful  enough  to  handle 
die  context  switching  required  for  MILS, 
while  still  maintaining  system  performance. 

In  the  last  15  years,  the  number  of  con¬ 


text  switches  per  unit  of  time  that  a  state- 
of-the-art  microprocessor  can  handle  has 
increased  by  a  factor  of  1,000.  Processor 
speed  has  increased  by  a  factor  of  100.  The 
number  of  transistors  per  cubic  inch  on  a 
wafer  has  increased  by  a  factor  of  125.  We 
can  now  perform  50,000  context  switches 
at  a  cost  of  5  percent  of  the  microproces¬ 
sor  clock.  This  5  percent  is  the  MILS  secu¬ 
rity  and  safety  tax. 

Because  information  originates  only 
from  authorized  sources,  is  delivered  only 
to  the  intended  recipients,  and  the  source  is 
authenticated  to  that  recipient,  die  applica¬ 
tion  developer  is  empowered  to  build  his  or 
her  own  reference  monitors2  at  the  applica¬ 
tion  layer  that  include  die  following: 

•  Non-bypassable.  Security  functions 
cannot  be  circumvented. 

•  Evaluatable.  Security  functions  are 
small  and  simple  enough  to  enable  rig¬ 
orous  proof  of  correctness  through 
mathematical  verification. 

•  Always  Invoked.  Security  functions 
are  invoked  each  and  every  time. 

•  Tamperproof.  Security  functions  and 
their  data  cannot  be  modified  without 
authorization,  either  by  subversive  or 
poorly  written  code. 

An  acronym  for  these  four  attributes  is 
NEAT  [9],  Security  policy  enforcement 
that  is  not  NEAT  is  not  effective.  Although 
other  operating  systems  have  offered  some 
form  of  non-bypassability  and  tamper¬ 
proof  functionality,  MILS  provides 


Figure  1 :  MILS  Architecture  Information  Flows 


Common  Terms 

API 

Application  Programming 

MMU 

Memory  Management  Unit 

Interface 

MSLS 

Multiple  Single  Levels  of  Security 

CIK 

Crypto  Ignition  Key 

PCS 

Partitioning  Communications 

CORBA 

Common  Object  Request 

System 

Broker  Architecture 

SOAP 

Simple  Object  Access  Protocol 

DDS 

Data  Distribution  Service 

RTOS 

Real-Time  Operating  System 

EAL 

Evaluation  Assurance  Level 

TCP/IP 

Transport  Control  Protocol/ 

HAL 

Hardware  Abstraction  Layer 

Internet  Protocol 

HTTP 

Hyper-Text  Transfer  Protocol 

UDDI 

Universal  Description,  Discovery, 

IPv6 

Internet  Protocol  version  6 

and  Integration 

MILS 

Multiple  Independent  Levels  of 

WSDL 

Web  Services  Description 

Security 

Language 

MLS 

Multiple  Levels  of  Security 

XML 

Extensible  Markup  Language 

August  2005 


www.stsc.hill.af.mil  1 3 


Systems:  Fielding  Capabilities 


Figure  2:  Secure  Network,  System  Configuration 


NEATness  for  die  first  time  in  a  COTS 
package  that  is  formally  modeled  and  math¬ 
ematically  verified  at  a  high  assurance  level. 

Divide  and  Conquer 

The  duration,  schedule  risk,  and  cost  of 
evaluating,  certifying,  and  deploying  soft¬ 
ware  increase  non-linearly  with  the  size  of 
the  code.  These  increases  are  especially 
onerous  at  high  levels  of  assurance.  Guar¬ 
anteed  NEATness  enables  us  to  design  a 
MLS  or  Multiple  Single  Levels  of  Security 
(MSLS)3  system  as  a  set  of  independent 
system  high  partitions  with  cross-domain 
servers,  downgraders,  and  guards  enabling 
secure  communications  both  among  those 
partitions  and  also  with  external  systems. 

Another  MILS  objective  is  to  enable 
the  evaluation  and  certification  of  a  com¬ 
plex  system  to  be  broken  down  into  a  num¬ 
ber  of  independent,  small  evaluations. 
Security-critical  software  components  that 
handle  more  than  one  level  of  information 
can  be  evaluated  at  high  levels  of  robust¬ 
ness,  approximately  Evaluation  Assurance 
Level  (EAL)  6+  of  die  Common  Criteria, 
an  internationally  approved  set  of  security 
standards  [10].  Cross-domain  servers, 
downgraders,  and  guards,  leveraging 
NEATness,  can  be  small  and  tightly 
focused,  making  high-assurance  evalua¬ 
tions  of  those  components  practical, 
achievable,  and  affordable.  Single-level  par¬ 
titions,  which  each  deal  with  only  one  level 
of  information,  can  be  evaluated  at  medi¬ 
um  levels  of  robustness,  approximately 
EAL  4,  which  is  practical  and  achievable 
for  large  bodies  of  code.  The  independ¬ 
ence  of  these  evaluations  also  enables 
reuse  of  code,  reuse  of  application  pro¬ 
gramming  interfaces  (APIs),  reuse  of  spec¬ 
ifications,  reuse  of  evaluation  artifacts,  and 
reuse  of  certifications  to  the  greatest 
degree  possible. 

Connecting  to  Other  Systems 

MILS  network  components  such  as  proto¬ 
col  stacks  and  their  associated  interface 
device  drivers  can  be  put  into  partitions  of 
their  own.  This  architecture  has  several 
advantages: 

•  Network  facilities  can  be  used  by  multi¬ 
ple  application  partitions. 


•  Network  data  is  processed  in  unprivi¬ 
leged  user  mode,  eliminating  a  vulnera¬ 
bility  that  is  a  common  avenue  of 
attack. 

•  Complex  protocol  code  such  as 
Internet  Protocol  (IP)  Ver.  6  can  be 
evaluated  and  certified  independent  of 
the  applications  using  die  code,  en¬ 
abling  reuse  of  the  evaluation  artifacts. 
Applications  use  an  API  to  interact  with 

the  network.  The  MILS  network  API  can 
have  the  same  semantics  as  in  a  traditional 
operating  system  such  as  the  familiar 
Transport  Control  Protocol/ IP  socket  calls. 
The  API  implementation  difference  can  be 
completely  under  the  hood. ,  transparent  to  the 
application  developer.  Instead  of  interact¬ 
ing  directly  with  die  protocol,  a  MILS  sock¬ 
et  implementation  uses  the  separation  ker¬ 
nel’s  interpartition  communications  facility 
to  forward  outgoing  data  to  the  protocol 
stack.  Incoming  data  is  handled  similarly  in 
the  opposite  direction. 

Secure  Network  Systems 

The  network  is  the  platform.  The  embed¬ 
ded  computer  that  is  not  connected  to 
another  processor  is  a  rare  exception.  By 
enforcing  its  four  foundational  security 
policies,  MILS  implements  a  robust  infor¬ 
mation  assurance  foundation  in  a  single 
node.  We  can  then  implement  a  robust 
information  assurance  foundation  through¬ 
out  a  distributed  system  by  providing  end-to- 
end  enforcement  of  those  same  security 
policies.  End-to-end  enforcement  is  pro¬ 
vided  by  a  high-assurance  middleware  com¬ 
ponent  called  the  Partitioning  Communi¬ 
cations  System  (PCS).  Leveraging  the  sepa¬ 
ration  kernel’s  guarantee  of  controlled 
information  flow  within  a  single  node,  the 
PCS  is  always  interposed  between  an  appli¬ 
cation  and  the  protocols/ drivers  that  effect 
an  off-board  data  transfer.  The  configura¬ 
tion  is  illustrated  in  Figure  2. 

The  PCS  enforces  the  security  policies 
end-to-end  by  providing  the  following: 

•  Strong  identity  of  each  node  within  a 
collection  of  MILS  nodes  (an  enclave). 

•  Separation  by  level  and/or  community 
of  interest.  Enclaves  are  then  connect¬ 
ed  together  via  high-assurance  MILS 
cross-domain  servers. 

•  Secure  configuration,  validating  that  all 
security  databases  are  consistent. 

•  Secure  image  loading. 

•  Secure  clock  synchronization. 

•  Provisioning  of  bandwidth  and  quality 
of  service. 

•  Suppression  of  covert  channels. 

Network  Middleware  Tools 

While  we  are  developing  new  systems  that 


enable  the  warfighter  to  share  information, 
it  is  important  to  not  reinvent  the  wheel. 
Distributed  system  solution  designers  make 
frequent  use  of  COTS  network  middleware 
for  various  application  paradigms: 

•  Client/server,  often  using  CORBA  [1 1], 
distributes  logic. 

•  Publish/subscribe,  often  using  DDS, 
distributes  data. 

•  Web-enabled  services,  using  Hyper¬ 
Text  Transfer  Protocol  servers  and 
components  such  as  Extensible  Markup 
Language;  SOAP  [Simple  Object 
Access  Protocol];  Web  Services 
Description  Language;  Disco;  Universal 
Description,  Discovery  and  Integration; 
etc.  that  enable  the  communication 
between  large  diverse  distributed  com¬ 
munities  of  interest. 

All  of  these  technologies  can  be  viewed 
as  tools  that  provide  the  application  pro¬ 
grammer  with  a  higher  level  abstraction  to 
the  rudimentary  socket  interface.  Much  of 
the  code  for  these  networking  middleware 
technologies  can  either  reside  together  in 
the  same  partition  as  the  application  that  it 
supports,  or  it  can  reside  in  a  partition  by 
itself.  In  either  case,  porting  existing  code 
to  a  MILS  environment  is  a  straightforward 
task  because  the  socket  API  does  not  need 
to  change.  The  PCS  still  fits  between  the 
network  middleware  and  the  protocol  stack 
and/ or  device  drivers. 

The  system  architect  should  not  view 
the  use  of  CORBA,  DDS,  or  Web  services 
as  mutually  exclusive.  A  single  application 
can  use  CORBA  for  remote  invocation,  for 
distribution  of  logic,  and  for  smart  pull  of 
needed  information;  it  can  use  DDS  for 
smart  push  of  sensor  data  that  is  being  mon¬ 
itored;  and  it  can  use  Web  services  as  a 
graphical  interface  for  reports  from  one 
large  community  of  interest  to  another,  e.g., 
the  Army  infantry  reporting  threat  data  and 
the  Air  Force  monitoring  Web  reports  and 
providing  air  support. 

There  is  an  interesting  side  benefit  to 
combining  network  middleware  with  the 
PCS.  One  of  the  purposes  of  network 
middleware  is  to  make  the  number  and 
location  of  processors  sharing  traffic  as 
transparent  as  possible  to  the  application. 
At  the  same  time,  in  a  secure  networked 
system,  we  need  to  know  exactly  where 
our  data  came  from  and  where  it  will  go. 
Merging  PCS  functions  with  network  mid¬ 
dleware  such  as  CORBA  or  DDS  gives  the 
system  designer  the  flexibility  to  relocate 
system  functions  without  introducing  new 
threats  to  data  confidentiality  or  integrity. 
This  enables  ad  hoc  networks  and  coali¬ 
tions  to  be  formed  based  on  newly  identi¬ 
fied  threat  data,  and  to  be  dissolved  as 
soon  as  the  threat  is  dealt  with. 
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Security  is  required  when  fielding  sys¬ 
tems  that  are  either  mission-critical  or  use 
national  information.  At  die  same  time, 
there  is  a  massive  investment  in  applica¬ 
tions  using  traditional  operating  systems 
and  traditional  middleware.  The  MILS 
architecture  can  provide  a  high-assurance 
foundation  for  fielded  systems  while  pre¬ 
serving  much  of  die  legacy  code  base. 

Guest  Operating  Systems 

A  traditional  operating  system,  either  an 
embedded  real-time  operating  system  (e.g., 
INTEGRITY,  VxWorks,  or  LynxOS)  or  a 
desktop  operating  system  (e.g.,  Linux, 
Windows,  or  Solaris)  can  run  inside  a  MILS 
partition  as  a  guest  operating  system.  Operating 
systems  written  with  portability  in  mind 
have  a  hardware  abstraction  layer  (HAL) 
drat  localizes  all  processor-specific  func¬ 
tions.  Writing  a  new  HAL  is  the  major  task 
in  porting  to  a  new  central  processing  unit. 
You  can  use  that  same  expertise  to  write  a 
HAL  that  abstracts  die  MILS  separation 
kernel  as  die  hardware  to  die  guest. 

By  itself,  die  guest  operating  system 
concept  enables  legacy  applications  to  be 
easily  ported  to  die  high-assurance  MILS 
environment.  Another  possibility  is  that 
multiple  MILS  partitions  can  each  contain 
an  instance  of  the  guest  operating  system. 
This  effectively  creates  multiple  virtual 
operating  systems  on  a  single  real  micro¬ 
processor.  The  MILS  separation  kernel 
provides  trustworthy  separation  with  respect 
to  both  memory  access  and  central  pro¬ 
cessing  unit  time.  Communications  among 
die  partitions  is  limited  to  those  paths 
explicitly  created  when  die  system  was  gen¬ 
erated.  This  is  a  practical  path  to  imple¬ 
mentation  of  cross  domain  solutions.  It  is 
also  a  practical  path  to  implementation  of 
high-assurance  workstations  suitable  for 
MLS  or  coalition  force  operations. 

For  example,  inside  a  partition  the  guest 
operating  system  can  run  as  a  thin  client;  it 
can  be  downloaded  from  a  remote  server. 
Which  remote  server  a  thin  client  is  down¬ 
loaded  from  can  be  determined  from  a 
token  reader  or  crypto  ignition  key.  The 
token  would  indicate  nationality,  clearance, 
and  job  tide.  The  PCS  would  open  a  secure 
connection  to  a  server  that  the  user,  who 
inserted  and  unlocked  the  token,  was 
authorized  to  communicate  with.  Access  to 
die  local  devices  such  as  screen,  keyboard, 
mouse,  hard  drive,  etc.,  would  be  provided 
by  the  MILS  workstation. 

Supporting  the  Warfighter 

The  Air  Force  Research  Laboratory 
(Information  Directorate),  in  cooperation 
with  the  National  Security  Agency, 


Department  of  Defense  prime  contractors, 
academia,  and  software  suppliers,  is  manag¬ 
ing  a  MILS  program  to  combine  die  best  of 
existing  commercial  standards  for  flight 
safety  and  integrated  modular  avionics  with 
die  following: 

•  DO-178B,  Software  Considerations  in 
Airborne  Systems  and  Equipment 
Certification,  Level  A  [12], 

•  ARINC-653,  Avionics  Application 
Software  Standard  Interface  [13]. 

The  program  is  also  combined  with 
die  following  appropriate  standards  for 
security: 

•  Common  Criteria  (International  Or¬ 
ganization  for  Standardization  15408), 
EAL  6,  augmented  [10]. 

•  Director  of  Central  Intelligence  Direc¬ 
tive  6/ 3,  Protecting  Sensitive  Compart- 
mented  Information  Widiin  Informa¬ 
tion  Systems,  Protection  Level  5  [14]. 
There  is  significant  synergy  among 

these  standards.  While  they  each  have  a  spe¬ 
cific  area  of  interest,  there  is  a  great  deal  of 
common  ground  between  safety-critical 
and  security  standards  with  respect  to 
sound  engineering  practice,  meeting 
requirements,  and  having  die  plans  in  place 
to  address  flaws. 

The  participating  software  suppliers 
that  are  currently  developing  MILS  separa¬ 
tion  kernels  are,  alphabetically,  Green  Hills 
Software,  Inc.  [15],  Lynux Works,  Inc.  [16], 
and  Wind  River  Systems,  Inc.  [17]. 
Objective  Interface  Systems,  Inc.  [18]  is 
currently  developing  die  partitioning  com¬ 
munications  system. 

Putting  It  All  Together 

MILS  is  all  about  keeping  tilings  separate 
that  need  to  be  separate  and  doing  so  with 
components  that  we  can  trust  with  our 
most  important  data  under  the  most  severe 
threat.  For  security,  we  are  keeping  data 
separate  by  classification  level,  by  commu¬ 
nity  of  interest,  and  by  nationality.  For 
safety,  we  are  keeping  applications  separate 
by  level  of  criticality.  All  of  this  is  done 
with  COTS  software  and  certification  arti¬ 
facts  that  are  reusable.  Leveraging  this 
reusability  makes  MSLS/MLS  system 
development  practical  and  certification/ 
accreditation  affordable  and  achievable. 
The  end  result  is  fielded  systems  that  have 
high-assurance  foundations  but  do  not 
require  custom-built  security  architectures 
for  each  new  system. 

For  more  information  about  MILS, 
please  see  <http://mils.ois.com>. ♦ 
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Notes 

1 .  Orange  book  levels  began  -with  Al  and 
moved  down  through  levels  B3,  B2,  Bl, 
C2,  and  Cl. 

2.  A  reference  monitor  is  an  Access 
Control  concept  referring  to  an  abstract 
machine  that  mediates  all  accesses  to 
objects  by  subjects  [3], 

3.  MSLS  means  there  are  multiple  chan¬ 
nels  each  with  their  own  separate  data 
classification  [9]. 
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Six  Steps  to  a  Successful  COTS  Implementation 


Arlene  F.  Minkiewicz 
PRICE  Systems 

A  successful  implementation  of  a  commercial  off-the-shelf-intensive  software  system  can  save  programs  money  if  you  have  the 
right  solution  and  understand  the  potential  risks  involved. 


Federal  organizations  are  relying  more 
and  more  on  commercial  applications 
to  supplement,  enhance,  or  replace  propri¬ 
etary  systems.  This  dependency  is  driven 
by  the  promise  of  improved  functionality 
and  reduced  total  ownership  cost,  as  well 
as  concern  over  the  lack  of  capability  to 
develop  and  maintain  proprietary  infor¬ 
mation  technology  applications.  However, 
failure  to  successfully  select,  control,  and 
implement  these  critical  components  con¬ 
tinues  to  result  in  projects  that  are  deliv¬ 
ered  late  and  over-budget  or  that  fail 
entirely. 

The  following  six-step  methodology  high¬ 
lights  the  important  activities  that  should 
take  place  during  a  commercial  off-the- 
shelf  (COTS)  implementation.  Following 
dais  methodology  throughout  the  software 
development  life  cycle  will  ensure  that  sig¬ 
nificant  activities  are  not  being  ignored 
and  will  increase  the  chances  of  planning, 
executing,  and  deploying  a  successful 
COTS-based  software  solution. 

One  of  the  biggest  problems  sighted  in 
COTS-based  projects  is  a  disconnect  between  time 
and  cost  expectations  during  planning  and  those 
actually  realised. 

During  the  planning  stages,  it  is  impor¬ 
tant  to  plan  appropriately  for  all  the  major 
activities  necessary  to  devise  a  well 
thought-out  solution  that  will  not  fall 
apart  with  the  first  upgrade  of  one  of  its 
components.  Research  [1,  2,  3,  4,  5]  has 
indicated  the  essential  activities  that  must 
take  place  to  ensure  successful  COTS- 
based  projects: 

•  Analyze  software  requirements. 

•  Evaluate  and  select  COTS  solution(s). 

•  Negotiate  terms  with  COTS  vendor. 

•  Implement  the  COTS-based  solution. 

•  Maintain  and  upgrade  the  COTS- 

based  solution. 

•  Maintain  license,  subscription,  and 

royalty  fees. 

Figure  1  portrays  an  overview  of  the 
six  steps  outlined  above  and  highlights  the 
interactions  that  may  occur  throughout 
the  execution  of  these  steps.  While  this 
diagram  implies  a  time  dependency 
between  these  steps,  it  is  important  to 
realize  that  in  certain  cases  this  is  neither 


strictly  adhered  to  nor  are  all  the  steps 
necessarily  performed  by  the  organiza¬ 
tion^)  contracted  to  deliver  a  solution. 
Some  requirements  analysis  and  COTS 
evaluation  are  likely  to  occur  in  very  early 
stages  of  a  project  as  feasibility  and 
affordability  are  analyzed. 

The  following  sections  provide  more 
details  about  each  of  these  steps,  along 
with  a  brief  description  of  the  factors  to 
consider  when  evaluating  the  affordability 
and  timeliness  of  a  COTS-based  solution. 
Specifics  about  the  quantification  and 
application  of  these  factors  can  be  found 
in  [6], 

I. Analyze  Software 
Requirements 

Software  requirements  analysis  is  a  critical 
part  of  the  software  development  process, 
although  too  often  this  activity  is  over¬ 
looked  or  glossed  over  in  the  rush  to  start 
building  software.  The  requirements  analysis 
process  is  necessary  to  determine  what 
functionality  is  necessary  to  deliver  the 
capability  required  by  the  eventual  end- 
user(s). 

There  are  two  general  areas  that  need 
to  be  explored  when  determining  and 
documenting  requirements  for  a  software 
system:  end  user  requirements  and  techni¬ 
cal  requirements.  The  discovery  process 
for  end-user  requirements  involves  busi¬ 
ness  analysts  or  requirements  engineers 
asking  the  end-user  what  they  expect  from 
the  software.  Once  end-user  requirements 
have  been  gathered,  an  important  next 
step  is  for  the  business  analysts  or  require¬ 
ments  engineers  to  restate  those  require- 

Figure  1 :  Overview  of  the  Six  Steps 


Negotiate 

Terms 
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COTS  fall  short  of  expectations? 


ments  and  present  diem  back  to  the  end- 
user  to  ensure  proper  understanding. 
Technical  requirements  can  be  gathered 
through  discussions  with  engineers  who 
understand  the  technical  nature  of  the 
problem  being  solved. 

The  question  of  whether  or  not  COTS 
solutions  are  a  viable  alternative  becomes 
an  important  factor  during  the  software 
requirements  analysis  activity  because  the 
software  requirements  drive  the  selection 
criteria  for  potential  COTS  solutions.  This 
being  said,  it  is  also  important  not  to  let 
the  availability  of  COTS  solutions  cloud 
the  analysis  by  obscuring  requirements. 

Solution  providers  should  be  aware 
that  it  is  unlikely  that  any  COTS  solution 
will  be  available  to  satisfy  all  the  require¬ 
ments  for  a  software  system.  The  require¬ 
ments  analysis  process  should  identify 
which  requirements  are  the  must  have 
requirements  and  which  can  be  somewhat 
bendable.  During  the  evaluation,  and  possi¬ 
bly  during  implementation,  tradeoffs  will 
be  necessary  to  compensate  for  function¬ 
ality  not  available  in  COTS  solutions  (or 
promised  and  not  delivered  with  a  chosen 
COTS  solution).  Decisions  should  be 
made  during  the  requirements  analysis 
activity  to  determine  which  functions  can 
be  subject  to  such  tradeoffs  and  which 
cannot.  If  most  or  all  of  the  software 
requirements  are  determined  to  be  must 
haves,  it  is  wise  to  revisit  the  decision  to 
pursue  a  COTS-intensive  solution. 

Although  the  focus  of  this  article  is  on 
COTS  software  solutions,  it  is  important 
to  mention  here  that  when  entire  systems 
are  being  constructed,  the  COTS  decision 
may  need  to  be  visited  even  before  soft- 
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ware  requirements  analysis  commences. 
During  die  analysis  of  system  require¬ 
ments,  decisions  may  be  required  to  deter¬ 
mine  whether  certain  functionality  should 
be  addressed  with  hardware  or  software. 
The  availability  of  software  COTS  solu¬ 
tions  could  be  a  factor  in  the  determina¬ 
tion  of  the  affordability  of  such  tradeoffs. 

As  with  all  activities  in  a  software 
development  process,  successful  execu¬ 
tion  of  the  requirements  analysis  process 
takes  time  and  effort.  The  major  driver  in 
determining  time,  cost,  or  effort  for  the 
software  requirements  analysis  activity  is  a 
measure  of  the  amount  of  functionality 
die  software  system  is  intended  to  deliver. 
The  measurement  of  software  size  is  always 
a  challenge.  A  measure  that  quantities 
functionality  delivered  such  as  function 
points  or  analogies  to  known  systems  is 
best. 

Whether  the  plan  is  for  that  function¬ 
ality  to  be  delivered  primarily  with  COTS 
solutions,  newly  developed  solutions,  or 
some  combination  of  the  two,  the  time 
and  effort  devoted  to  requirements  analy¬ 
sis  should  be  fairly  consistent.  The  techni¬ 
cal  complexity  of  die  functionality  as  well 
as  die  software’s  operational  platform  will 
also  drive  the  requirements  analysis  effort 
because  more  complex  solutions  require 
more  time  to  understand  and  communi¬ 
cate.  Additionally,  die  presence  of  project 
constraints,  timing,  memory,  or  schedule 
will  impact  the  effort  required  for  this 
activity. 

2.  Evaluate  and  Select  COTS 
Solution(s) 

Once  a  decision  to  pursue  a  COTS  alter¬ 
native  is  made,  the  first  step  is  to  deter¬ 
mine  the  availability  of  COTS  solutions 
that  have  the  potential  to  provide  needed 
functionality,  then  evaluate  these  solu¬ 
tions.  The  main  reasons  to  evaluate  are  the 
following: 

•  Determine  whether  die  functionality 
promised  is  the  functionality  delivered. 

•  Determine  whedier  system  non-func¬ 
tional  requirements  (portability,  relia¬ 
bility,  security,  performance)  can  be 
met. 

•  Determine  whedier  functional  require¬ 
ments  can  be  met  by  the  functionality 
delivered. 

•  Determine  whether  a  proposed  suite 
of  components  can  operate  success¬ 
fully  in  the  environment(s)  where  the 
system  is  intended  to  operate. 

•  Determine  whether  a  proposed  set  of 
components  can  operate  successfully 
in  an  integrated  fashion. 

•  Determine  die  stability  and  viability  of 


die  vendor. 

•  Determine  the  willingness  of  the  ven¬ 
dor  to  cooperate  and  help  make  the 
project  successful. 

The  evaluation  needs  to  be  focused  on 
more  than  just  product  characteristics 
such  as  functionality,  maturity,  technology, 
architecture,  and  long-term  viability. 
There  should  also  be  a  focus  on  vendor 
characteristics  such  as  maturity,  stability, 
cooperation,  and  ability  to  provide  ade¬ 
quate  support,  training,  and  documenta¬ 
tion.  The  evaluation  should  also  be  used 
to  ensure  that  there  are  no  compatibility 
issues  associated  with  using  COTS  solu¬ 
tions  from  multiple  vendors. 

The  selection  process  is  often  a  com¬ 
bination  of  the  following  three  evaluation 
techniques: 

•  Progressive  filtering  of  available 
COTS  components.  This  requires 
several  iterations  of  filtering,  each  one 

“When  performing  a 
COTS  evaluation,  it  is 
valuable  to  obtain 
references  from  the 
COTS  vendors  in  an 
effort  to  speak  with 
developers  and  end 
users  who  have 
worked  with  a  particular 
COTS  solution.  ” 


going  into  progressively  more  detail 
until  a  single  solution  or  set  of  solu¬ 
tions  emerges  as  the  best  answer. 

•  Puzzle  assembly  process.  This 
approach  suggests  that  it  is  better  to 
evaluate  a  set  of  components  at  one 
time  using  an  evolutionary  prototyping 
approach.  In  diis  method,  multiple  sets 
may  be  evaluated  in  parallel  to  identify 
which  of  them  comes  closest  to  solv¬ 
ing  the  entire  pusgle  presented  by  the 
system  requirements. 

•  Identifying  the  keystone  COTS 
software  components.  This  is  identi¬ 
fying  those  components  for  which 
requirements  (whether  they  be  techni¬ 
cal,  process,  functional,  vendor,  etc.) 
are  unbendable,  and  then  basing  other 
component  selections  on  compatibility 
and  ease  of  interface  with  die  keystone 


components. 

A  commonly  cited  challenge  by  system 
integrators  is  that  COTS  products  often 
fail  to  deliver  die  functionality  or  other 
requirements  promised  during  evalua¬ 
tions.  It  is  prudent  to  get  some  hands-on 
time  widi  the  components  being  evaluated 
where  possible;  this  may  be  problematic  as 
it  most  likely  requires  a  great  deal  of  coop¬ 
eration  and  support  from  both  the  vendor 
and  the  integrating  organization. 

When  performing  a  COTS  evaluation, 
it  is  valuable  to  obtain  references  from  the 
COTS  vendors  in  an  effort  to  speak  with 
developers  and  end  users  who  have 
worked  with  a  particular  COTS  solution. 
This  not  only  provides  valuable  feedback 
about  vendors’  responsiveness  and 
dependability,  it  also  aids  the  planning 
process  by  highlighting  problems  or  pit- 
falls  odier  integrators  may  have  experi¬ 
enced. 

The  evaluation  and  selection  activity 
not  only  facilitates  identification  of  avail¬ 
able  COTS  solutions,  it  also  points  to 
those  pieces  of  functionality  that  cannot 
be  satisfactorily  implemented  by  existing 
off-the-shelf  solutions.  An  important 
byproduct  of  dais  investigation  may  be  an 
examination  of  the  cost,  schedule,  and 
effort  associated  with  developing  custom 
code  to  make  up  for  required  functionali¬ 
ty  missing  in  COTS  solutions.  This  exam¬ 
ination  may  require  revisiting  the 
cost/benefit  analysis  leading  to  the  deci¬ 
sion  to  build  a  COTS-based  solution. 

Generally,  the  time  and  effort  devoted 
to  the  selection  and  evaluation  activity  is 
often  based  on  a  predetermined  level  of 
effort.  The  determination  of  this  level  of 
effort  should  consider  the  amount  of 
functionality  being  implemented  with 
COTS  solutions  (based  on  functional  size 
or  analogy),  die  number  of  solutions  that 
will  be  evaluated,  the  type(s)  of  evalua¬ 
tions  being  performed,  and  the  number 
and  criticality  of  evaluation  criteria. 

3.  Negotiate  Terms  With 
COTS  Vendors 

Certainly  it  is  important  to  negotiate  the 
best  deal  possible  when  working  with  one 
or  more  vendors  to  craft  a  solution.  It  is 
even  more  important  to  understand  the 
impact  of  these  negotiations  and  their 
timing  on  the  eventual  success  or  failure 
of  your  project.  Several  of  the  most  com¬ 
monly  cited  challenges  of  those  building 
software  solutions  with  COTS  compo¬ 
nents  involve  vendor  forthrightness  and 
cooperation  [4], 

Vendors  are  much  more  likely  to 
address  customer  concerns  with  missing 
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or  incomplete  functionality  and/or  bugs 
in  the  software  before  signing  on  the  dot¬ 
ted  line.  During  the  negotiation  process,  it 
is  important  to  address  and  resolve  any 
known  issues  and  establish  expectations 
for  issues  that  emerge  during  the  integra¬ 
tion  process  and  throughout  the  product 
life  cycle.  Clearly,  the  size  of  die  vendor 
and  the  size  of  the  purchase  may  be  fac¬ 
tors  in  determining  how  demanding  any 
particular  customer  can  be,  but  it  is  impor¬ 
tant  to  set  expectations  with  all  of  the 
project  stakeholders. 

The  end  result  of  this  negotiation 
should  be  a  clear  picture  of  the  non-recur¬ 
ring  and  recurring  costs  associated  with 
die  system  being  developed.  A  nonrecur¬ 
ring  cost  is  a  one-time  fee  associated  with 
acquiring  the  COTS  solution  such  as  the 
purchase  price  of  shrink-wrapped  soft¬ 
ware.  Recurring  costs  are  those  generally 
based  on  usage  or  time  of  use  such  as 
annual  licensing  fees.  Negotiations  should 
also  result  in  a  common  understanding 
between  parties  of  update  and  upgrade 
policies,  as  well  as  expectations  with 
regard  to  vendor  responsiveness  and 
cooperation.  It  may  also  be  necessary  dur¬ 
ing  this  step  to  develop  a  plan  with  the 
vendor  to  ensure  that  maintenance  of  the 
deployed  solution  be  possible  even  if  the 
vendor  goes  out  of  business.  This  plan 
generally  involves  the  escrowing  of  source 
code  to  be  made  available  only  under 
terms  of  the  agreement  such  as  bankrupt¬ 
cy  or  company  demise. 

The  cost  and  effort  drivers  for  this 
activity  should  be  broken  into  two  parts. 
The  first  part  is  the  actual  dollar  value  that 
is  determined  for  delivery  of  die  COTS 
component  after  negotiations  and  any 
promised  royalties  and  other  fees.  The 
second  part  relates  to  the  amount  of  time 
and  resources  that  must  be  devoted  to  the 
negotiation  with  die  vendor. 

4.  Implement  the 
COTS-Based  Solution 

Once  analysis,  evaluation,  and  selection  of 
a  COTS-based  solution  are  complete, 
implementation  can  commence.  The  fol¬ 
lowing  activities  may  be  required  to  ensure 
successful  implementation. 

Tailoring  of  a  COTS  Solution 

There  are  certain  necessities  that  should 
be  performed  in  or  around  software  to  get 
die  COTS  software  components  config¬ 
ured  for  the  system  and  its  requirements. 
Databases  and  other  parameters  need  to 
be  initialized  and  loaded,  all  or  part  of  the 
components  need  to  be  registered  with 
die  operating  system,  security  must  be 


activated  or  initialized,  screens  and  reports 
need  to  be  scripted,  and  other  script  devel¬ 
opment  may  be  required. 

Although  these  activities  are  unlike  tra¬ 
ditional  coding  exercises,  they  do  take  time 
and  effort  to  complete.  The  results  of 
diese  activities  require  testing  and  verifica¬ 
tion.  These  tasks  also  require  a  significant 
level  of  understanding  as  to  how  the 
COTS  component(s)  work  and  how  to 
work  with  diem.  This  requires  reading 
manuals  and/ or  attending  training  and 
dien  experimenting  with  the  actual  com¬ 
ponents  to  reach  a  level  of  competency. 

The  major  cost,  effort,  and  schedule 
drivers  for  the  tailoring  activity  include  the 
amount  and  complexity  of  scripts,  data¬ 
base  parameters,  reports,  and  screens 
being  customized.  Additionally,  as  security 
and  access  control  requirements  increase 
in  rigor,  they  will  drive  up  time  and  cost.  It 
is  important  also  to  consider  the  ease  with 
which  the  COTS  solution  can  be  under¬ 
stood,  the  quality  of  training  and  docu¬ 
mentation,  and  the  integration  team  famil¬ 
iarity  with  die  COTS  solutions  being  used 
and  die  system  being  implemented. 

Modification  of  COTS  Software 

Generally  the  definition  of  COTS  soft¬ 
ware  precludes  modifications  because 
COTS  software  does  not  have  source 
code  available.  This  is  the  case  when  the 
COTS  software  is  a  shrink-wrapped  com¬ 
mercial  product.  Sometimes  solutions  call 
for  the  integration  of  a  series  of  off-the- 
shelf  components  that  do  not  meet  this 
traditional  definition  of  COTS  but  radier 
are  components  with  source  code  avail¬ 
able  diat  are  either  furnished  by  the  cus¬ 
tomer  or  otherwise  obtained.  While  these 
projects  are  not  strictly  COTS  projects, 
diey  do  happen  and  are  mentioned  here 
for  completeness. 

It  would  be  nice  if  the  COTS  software 
completely  satisfied  all  the  functional 
requirements  it  was  selected  to  meet,  but 
diis  is  often  not  die  case.  In  situations 
where  die  source  code  can  be  made  avail¬ 
able,  the  project  team  needs  to  make  a 
determination  whether  or  not  modifica¬ 
tion  is  an  option.  It  is  generally  a  bad  idea 
to  make  modifications  to  COTS  software 
because  diis  negates  much  of  the  produc¬ 
tivity  increase  obtained  from  using  COTS 
components  and  is  likely  to  jeopardize  any 
likelihood  of  supplier  maintenance  of  the 
COTS  components.  If  COTS  modifica¬ 
tions  are  being  considered,  care  should  be 
taken  to  ensure  that  these  modifications 
are  very  modular  in  nature.  The  develop¬ 
ers  need  to  learn  a  great  deal  about  the 
architecture  and  basic  structure  of  the 
solution  before  any  modifications  can  be 


made.  It  is  important  to  understand  that 
the  project  productivity  hit  can  be  sub¬ 
stantial  even  when  the  slightest  modifica¬ 
tions  are  made  to  a  COTS  component. 

The  major  cost,  effort,  and  schedule 
drivers  for  modifications  to  COTS  soft¬ 
ware  are  the  same  factors  that  drive  costs 
for  any  software  modification  project 
(functional  size,  extent  of  modification, 
technical  complexity,  eventual  operating 
platform,  productivity,  and  efficiency  of 
development  organization).  These  factors 
must  then  be  augmented  to  account  for 
unfamiliarity  of  die  COTS  solution  code, 
quality  of  COTS  training  and  documenta¬ 
tion,  vendor  cooperation,  development 
team  experience  widi  COTS  solutions,  and 
the  COTS  integration  process.  It  is  also 
important  to  include  maintenance  of  die 
entire  COTS  component  in  the  affordabil¬ 
ity  analysis  as  once  modifications  have  been 
made  it  is  unlikely  drat  the  COTS  vendor 
will  continue  to  support  dieir  solution. 

Design,  Code,  and  Test  of  Glue  Code 

Glue  code  literally  holds  the  system 
together.  Glue  code  is  any  code  that  needs 
to  be  written  to  make  the  COTS  software 
components  function  as  advertised 
and/ or  as  required.  It  is  dae  code  that  ref¬ 
erences  dae  interfaces  in  dae  COTS  soft¬ 
ware  component  and  needs  to  interpret 
return  codes  from  daese  interfaces.  Glue 
code  is  often  required  to  convert  data  and 
other  information  from  the  format  in 
which  the  system  maintains  data  to  the 
format  required  by  the  COTS  component. 
In  a  well-written  application,  the  glue  code 
acts  as  a  layer  between  dae  system  and 
COTS  components,  encapsulating  the 
data  in  such  a  way  that  upgrades  and 
replacements  are  as  painless  as  possible. 
Finally,  glue  code  is  sometimes  required  to 
add  functionality  that  the  COTS  software 
component  implements  inadequately  or 
that  should  be  provided  by  the  COTS 
component  but  is  not. 

These  development  activities  are  com¬ 
plicated  due,  primarily,  to  unavailable 
source  code.  The  complexity  of  glue  code 
development  is  akin  to  a  situation  where 
an  entirely  new  team  of  developers  is 
brought  in  to  integrate  custom  built  com¬ 
ponents,  but  is  given  limited  or  no  access 
to  the  original  development  team  and  the 
source  code.  The  unfamiliarity  of  the 
interfaces,  along  with  the  inability  to 
debug  die  components,  adds  complexity 
to  die  development  exercise. 

Another  factor  diat  makes  glue  code 
development  differ  from  traditional  code 
development  is  the  complete  reliance  on 
vendors  to  fix  bugs  when  diey  are  discov¬ 
ered,  ensure  that  upgrades  are  upwardly 
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compatible,  release  stable  products,  and  fill 
in  where  documentation  falls  short.  Bug 
fixing  can  be  particularly  problematic  in  a 
COTS  project,  especially  when  there  are 
multiple  vendors  involved  or  when  modifi¬ 
cations  have  been  made  to  die  COTS  com¬ 
ponents.  Widi  multiple  vendors,  especially 
with  unavailable  source  code,  the  integra¬ 
tor  must  rely  on  the  vendors  not  only  to  fix 
die  bugs  but  also  to  cooperate  widi  each 
odier  in  the  identification  of  where  a  bug 
actually  resides.  When  the  COTS  compo¬ 
nents  have  been  modified,  the  integrator 
often  must  struggle  to  convince  the  ven¬ 
dor  that  the  bug  is  in  the  original  code  and 
not  a  side  effect  of  changes  they  have 
made.  Careful  modularization  and  docu¬ 
mentation  of  any  COTS  modifications 
may  help  alleviate  this  problem. 

As  with  modifications  to  COTS  com¬ 
ponents,  the  major  cost,  effort,  and  sched¬ 
ule  drivers  for  glue  code  development  are 
not  unlike  those  drivers  associated  with 
any  custom  development,  but  the  produc¬ 
tivity  of  the  development  team  needs  to 
be  adjusted  to  account  for  unfamiliarity 
and  unavailability  of  COTS  components 
along  with  the  requirement  for  vendor 
support  and  cooperation  in  solving  inte¬ 
gration  problems.  As  die  number  of  dif¬ 
ferent  COTS  components  and  the  num¬ 
ber  of  different  vendors  involved  increas¬ 
es,  so  increases  the  complexity  (thus  effort 
and  schedule)  of  this  activity. 

Integration  and  Test  of  COTS 
Components  With  Other  COTS  or 
Custom  Components 

The  system  needs  to  be  integrated  and 
tested  to  ensure  that  all  functional  and 
non-functional  requirements  are  met.  This 
activity  tests  the  entire  system  (or  subsys¬ 
tem)  as  a  complete  unit,  verifying  that 
there  are  no  system-level  problems  associ¬ 
ated  with  incompatibilities  or  competing 
demands  of  components  for  limited 
resources. 

Requirements  related  to  performance, 
reliability,  and  security  could  be  particular¬ 
ly  problematic  during  this  activity  as  these 
types  of  problems  are  likely  to  go  unno¬ 
ticed  until  the  whole  system  is  running 
together.  It  is  best  to  use  an  incremental  or 
evolutionary  approach  when  building 
COTS-inclusive  systems,  as  these 
approaches  advocate  successive  integra¬ 
tions  during  development  rather  than  a 
waterfall-type  process  where  integration 
does  not  happen  until  die  end  of  the 
development. 

A  well-designed  integration  and  test 
approach  would  focus  early  releases  on 
those  areas  that  are  high  risk  or  most  like¬ 
ly  to  lead  to  incompatibilities.  If  multiple 
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COTS  components  from  multiple  ven¬ 
dors  are  being  integrated,  it  is  important 
to  have  all  of  them  running  together  at  the 
same  time  as  early  as  possible,  even  if  each 
is  functioning  in  very  limited  capacity  with 
respect  to  its  intended  feature  set.  This 
helps  identify  potential  contention  and 
incompatibilities  between  components. 

When  assessing  die  effort  and  sched¬ 
ule  for  glue  code  development,  tailoring, 
or  integration  activities,  an  important  fac¬ 
tor  is  die  update  cycle  for  the  COTS  solu¬ 
tions  being  integrated.  If  updates/ 
upgrades  are  frequent,  additional  time  and 
effort  may  be  required  to  evaluate  and 
possibly  incorporate  these  updates. 


“It  is  generally  a  bad 
idea  to  make 
modifications  to  COTS 
software  because  this 
negates  much  of  the 
productivity  increase 
obtained  from  using 
COTS  components  and  is 
likely  to  jeopardize  ... 
supplier  maintenance  ...” 


Upgrades  and  updates  may  be  included  if 
diey  contain  required  bug  fixes,  improve¬ 
ments  to  keep  the  look  and  feel  current 
with  user  expectations,  or  features  that 
relate  to  missing  or  incomplete  require¬ 
ments  in  earlier  versions.  Vendor  quality 
and  stability,  along  with  vendor(s)  regular 
release  schedules,  are  important  factors  in 
assessing  effort  associated  widi  updates 
and  upgrades. 

In  general,  the  effort,  cost,  and  sched¬ 
ule  for  many  system-level  integration 
activities  is  likely  to  be  higher  for  COTS- 
inclusive  software  than  software  that  is 
composed  of  custom-built  components 
because  of  die  unfamiliarity  with  the  code. 
The  COTS  software  component  should 
be  viewed  by  the  integrator  as  a  black  box. 
The  factors  that  drive  the  effort  and 
schedule  for  integration  activities  include 
die  amount  of  functionality  being  inte¬ 
grated  (including  functionality  provided 
by  homegrown  development  as  well  as 
that  coming  from  the  COTS  compo¬ 
nents),  die  complexity  of  the  functionali¬ 


ty,  die  operating  platform  for  the  system, 
glue  code  size,  and  vendor-related  issues 
such  as  cooperation,  support,  and  upgrade 
policies. 

5.  Maintain  and  Upgrade  the 
COTS-Based  Solution 

Once  die  software  is  deployed,  the  follow¬ 
ing  two  ongoing  activities  are  required  to 
keep  it  operational  and  keep  end-users 
happy. 

Evaluation  and  Inclusion  of  Updates 
and  Upgrades  From  the  Vendor 

There  are  countless  reasons  why  the  inclu¬ 
sion  of  updates  and  upgrades  are  desirable 
once  the  product  is  deployed.  If  there  are 
bugs  in  the  COTS  software  diat  affect  sys¬ 
tem  operation,  then  an  upgrade  is  obvi¬ 
ously  needed.  Beyond  this,  the  vendor 
should  be  upgrading  the  product  to  keep 
up  with  rapidly  changing  technology.  To 
maintain  a  software  system  that  meets 
market  expectations  widi  respect  to  per¬ 
formance,  look  and  feel,  operating  plat¬ 
forms,  etc.,  updates  of  the  COTS  software 
components  will  be  likely. 

Whether  including  an  update  or 
upgrade,  each  refresh  of  the  COTS  com¬ 
ponents  poses  potential  risk.  Assessment 
is  required  with  each  release  to  determine 
whether  it  makes  sense  to  include  it  in  the 
software  system.  Sometimes  upgrades 
change  the  interfaces  to  the  COTS  soft¬ 
ware  components  so  that  with  the 
upgrade,  existing  functionality  ceases  to 
work  correctly,  requiring  a  rewrite  of 
some  of  die  glue  code.  Vendors  find  that 
in  order  to  offer  new  features  or  to  keep 
up  with  technology,  they  must  change 
interfaces  and  databases  or  rework  func¬ 
tionality.  To  get  access  to  the  new  features 
and  technology  in  an  upgrade,  the  soft¬ 
ware  developer  may  be  forced  to  accept 
changes  he  or  she  does  not  want  or  need 
as  well  -  creating  additional  cost  with  no 
added  value  to  the  software  system. 

Every  upgrade  also  carries  with  it  the 
possibility  of  incompatibilities  with  the 
existing  software  system,  other  COTS 
software  components,  or  even  the  operat¬ 
ing  platforms  on  which  die  system  runs. 
For  this  reason,  each  upgrade  should 
include  a  repeat  of  system  operational 
testing  and  system  regression  testing  to 
ensure  diat  die  effects  on  system  opera¬ 
tion  are  understood  and  desirable.  While  it 
is  important  to  understand  all  of  this 
when  evaluating  whether  or  not  to 
upgrade,  it  is  also  important  to  understand 
that  there  is  a  risk  associated  with  failure 
to  upgrade.  At  some  point,  the  vendor  will 
discontinue  support  of  older  versions  of 
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die  COTS  software. 

The  major  factors  that  drive  cost, 
effort,  and  schedule  for  die  evaluation  of 
COTS  upgrades  include  the  amount  of 
functionality  delivered  by  COTS  solutions, 
die  number  of  COTS  solutions  in  the  sys¬ 
tem,  the  upgrade /update  frequency  of  the 
vendor(s),  and  the  quality  and  stability  of 
die  COTS  solutions.  When  (and  if)  a  deci¬ 
sion  has  been  made  to  include 
upgrade/updates  of  COTS  solution(s), 
die  drivers  for  the  integration  and  test  are 
die  same  as  those  for  integration  and  test 
during  development,  although  the  extent 
of  effort  should  be  tempered  by  die 
extent  of  functionality  changes  in  the 
upgrades/ updates. 

Bug  Fixes 

Software,  whether  it  is  homegrown  or 
acquired  externally,  is  likely  to  have 
defects.  Bug  fixing  efforts  for  COTS- 
intensive  systems  will  differ  significandy 
from  typical  repair  efforts.  Additionally, 
bugs  may  exist  in  die  COTS  software 
component  that  the  vendor  is  unable  or 
unwilling  to  fix  for  which  workarounds  in 
die  glue  code  need  to  be  developed. 

The  effort  associated  widi  bug  fixes 
for  COTS  software,  as  with  any  software, 
is  a  direct  function  of  the  quality  of  the 
initial  software  offering,  as  well  as  the 
quality  with  which  bugs  are  fixed.  This 
quality  is  generally  a  function  of  die 
amount  of  functionality,  the  complexity  of 
die  system,  and  the  development  process¬ 
es  and  practices  employed  by  the  initial 
software  developer.  For  COTS  software, 
some  of  these  factors  are  apparent  and 
some  are  hard  —  if  not  impossible  —  to 
find  out.  Also,  how  and  when  the  bugs  in 
die  COTS  component  get  fixed  is  for  the 
most  part  out  of  die  integrators’  control. 

These  factors  complicate  the  process 
of  planning  for  maintenance  of  COTS- 
based  systems.  Additionally,  die  mainte¬ 
nance  process  is  plagued  widi  the  same 
issues  cited  earlier  for  the  glue  code 
design,  integration,  and  test:  It  is  not 
always  obvious  where  die  bugs  actually 
occur.  Despite  the  hurdles  mentioned,  it  is 
still  possible  to  plan  proactively  for  the 
maintenance  of  a  COTS-based  system 
based  on  what  is  known.  Functional  size 
of  the  entire  system  (including  not  just 
COTS  components  but  homegrown  com¬ 
ponents  as  well),  system  complexity,  oper¬ 
ating  platform,  glue  code  size,  amount  of 
modification,  and  maintenance  team  pro¬ 
ductivity  can  be  used  to  baseline  the 
effort.  This  effort  should  then  be  adjusted 
for  the  loss  of  productivity  associated 
with  debugging  through  black  box  code 
and  interfacing  with  multiple  vendors. 


6.  Maintain  License, 
Subscription,  and  Royalty  Fees 

License  or  maintenance  fees  need  to  be 
paid  in  order  to  ensure  updates  and 
upgrades  as  well  as  continuing  support  of 
die  COTS  components.  It  is  important  to 
understand  vendor(s)  upgrade  policies.  It 
is  also  wise  to  do  a  long-term  analysis  of 
die  differences  between  annual  subscrip¬ 
tion  fees  (if  subscription  is  an  option)  ver¬ 
sus  paying  for  upgrades  on  an  individual 
basis.  This  analysis  should  include  upgrade 
policies,  vendor  stability,  and  frequency  of 
releases.  License  and  royalties  should  be 
an  important  part  of  the  initial  negotiation 
process.  Renewal  periods  are  an  opportu¬ 
nity  to  revisit  the  terms  of  the  negotiation 
to  determine  whether  the  vendor  is  meet¬ 
ing  support  and  upgrade  commitments. 

Conclusions 

A  well  thought-out  and  well-executed 
software  project  that  incorporates  one  or 
many  COTS  solutions  can  happen  more 
quickly  and  be  more  cost  effective  than 
the  same  system  implemented  with  cus¬ 
tom  developed  components.  Too  often, 
COTS  projects  are  not  thought  out  or 
planned,  running  on  the  incorrect 
assumption  that  every  COTS  solution  is  a 
small  integration  project  without  the 
issues  and  complexities  cited  above.  This 
way  of  thinking  leads  to  unrealistic  and 
poorly  managed  expectations,  resulting  in 
failed  projects.  These  types  of  failures 
occur  when  projects  fail  to  plan  for  or 
incorporate  the  additional  activities 
unique  to  COTS-intensive  developments. 
Following  this  six-step  methodology  will 
ensure  that  important  activities  and  deci¬ 
sion  points  are  properly  executed,  reduc¬ 
ing  many  of  the  risks  associated  with 
such  developments. ♦ 
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Performance-Based  Earned  Value 


PaulJ.  Solomon 

Northrop  Grumman  Integrated  Systems 

Performance-Based  Earned  Value9  (PBEl™)  is  an  enhancement  to  the  Earned  l  ralue  Management  Systems  (El  MS) 
standard  [1]).  PBEl ^  overcomes  the  standard’s  shortcomings  with  regard  to  measuring  technical  performance  and  quality 
because  it  is  based  on  standards  and  models  for  systems  engineering,  software  engineering,  and  project  management.  The  dis¬ 
tinguishing  feature  of  PBEl '  is  its  focus  on  the  customer  requirements.  PBEl r  provides  principles  and  guidance  for  cost- 
effective  processes  that  specify  the  most  effective  measures  of  cost,  schedule,  and  product  quality  peformance. 


Program  managers  (PMs)  expect  accu¬ 
rate  reporting  of  integrated  cost, 
schedule,  and  technical  performance  when 
the  supplier’s  Earned  Value  Management 
Systems  (EVMS)  complies  with  the 
EVMS  guidelines  in  the  American 
National  Standards  Institute  (ANSI)/ 
Electronic  Industries  Alliance  (EIA) 
Standard-748-A-1998.  However,  EVM 
data  will  be  reliable  and  accurate  only  if 
the  following  occurs: 

•  The  indicated  quality  of  the  evolving 
product  is  measured. 

•  The  right  base  measures  of  technical 
performance  are  selected. 

•  Progress  is  objectively  assessed. 

Using  EVM  also  incurs  significant 
costs.  However,  if  you  are  measuring  the 
wrong  things  or  not  measuring  the  right 
way,  than  EVM  may  be  more  costly  to 
administer  and  may  provide  less  manage¬ 
ment  value  [2] . 

EVMS  Shortcomings 

The  EVMS  standard  has  significant  short¬ 
comings  with  regard  to  standards  and 
models  for  systems  engineering  (SE),  soft¬ 
ware  engineering,  and  project  manage¬ 
ment.  Consequently,  there  is  no  assurance 
the  reported  earned  value  (EV)  is  based 


on  product  metrics  and  on  the  evolving 
product  quality  as  defined  by  the  stan¬ 
dards  and  models. 

First,  the  EVMS  standard  states  that 
EV  is  a  measurement  of  the  quantity  of 
work  accomplished  and  that  the  quality 
and  technical  content  of  work  performed 
are  controlled  by  other  processes.  A  PM 
should  ensure  that  EV  is  also  a  measure¬ 
ment  of  the  product  quality  and  technical 
maturity  of  the  evolving  work  products 
instead  of  just  the  quantity  of  work 
accomplished.  However,  a  Naval  Air 
Systems  Command  (NAVAIR)  organiza¬ 
tion  that  used  EVM  and  the  Team 
Software  Process™  to  accelerate  software 
process  improvement  concluded  that 
EVM  did  not  address  product  quality  and 
was  not  beneficial  at  the  higher  levels  of 
the  Capability  Maturity  Model®  (CMM®) 
for  Software  [3]. 

Second,  the  EVMS  principles  address 
only  the  project  work  scope.  EVMS 
ignores  the  product  scope  and  product 
requirements. 

Third,  EVM  is  perceived  to  be  a  risk 
management  tool.  However,  EVMS  was 
not  designed  to  manage  risk  and  does  not 
even  mention  the  subject. 

The  following  guidance  will  enable  a 


PM  to  use  Performance-Based  Earned 
Value®  (PBEVSM)  to  overcome  the  limita¬ 
tions  of  EVMS  and  provide  a  framework 
for  utilizing  PBEV  as  a  key  component  of 
project  planning,  measurement,  and  con¬ 
trol.  The  guidance  is  based  on  actual  proj¬ 
ect  experience  and  has  contributed  to  the 
success  of  software-intensive  programs, 
including  the  B-2  stealth  bomber. 

Department  of  Defense  Policy 

Compliance  with  SE  standards  will  sup¬ 
port  the  Department  of  Defense  (DoD) 
acquisition  policy  that  programs  will 
implement  SE  plans  (policy)  [4] .  The  DoD 
also  published  the  Defense  Acquisition 
Guidebook  (DAG)  and  the  Systems 
Engineering  Plan  (SEP)  Preparation 
Guide  (SEP  Guide)  to  provide  discre¬ 
tionary  best  business  practices  to  comple¬ 
ment  the  policy.  The  SEP  Guide  cites 
engineering  standards1  that  are  sources  of 
PBEV  [5],  Table  1  shows  pertinent  policy 
components  and  implementing  guidelines. 

Product  Metrics  and  Quality 

The  Institute  of  Electrical  and  Electronics 
Engineers  (IEEE)  1220  and  the  EIA  632 
have  similar  guidance  regarding  product 
metrics  and  quality.  Product  metrics  allow 
assessment  of  the  product’s  ability  to  sat¬ 
isfy  requirements  and  to  evaluate  the 
evolving  product  quality  against  planned 
or  expected  values.  Establishing  a  time- 
phased  product  quality  requirements  base¬ 
line  against  which  progress  can  be  meas¬ 
ured  normally  precedes  the  schedule  and 
budget.  An  exception  for  the  system  defi¬ 
nition  stage  of  the  systems  development 
life  cycle,  before  the  real  product  quality 
requirements  are  known,  is  discussed  later. 
Of  equal  importance  are  a  disciplined 
requirements  traceability  process  and  a 
requirements  traceability  database  [6]. 


®  Performance-Based  Earned  Value  is  registered  with  the 
U.S.  Patent  and  Trademark  Office  by  Paul  Solomon. 
PBEV  is  a  service  mark  of  Paul  Solomon. 

Capability  Maturity  Model  and  CMMI  are  registered  in 
the  U.S.  Patent  and  Trademark  Office  by  Carnegie 
Mellon  University. 

CMM  Integration  and  Team  Software  Process  are  service 
marks  of  Carnegie  Mellon  University. 


Table  1 :  Department  of  Defense  System  Engineering  Policy  and  Guides 


DoD  SE  Policy  and  Guides 

Policy 

DAG 

SEP 

Guide 

Develop  systems  engineering  plan. 

P 

4. 2. 3. 2 

1.0 

Event-driven  timing  of  technical  reviews. 

P 

4.5.1 

3.4.4 

Success  criteria  of  technical  reviews. 

P 

4.5.1 

3.4.4 

Assess  technical  maturity  in  technical  reviews. 

4.5.1 

3.4.4 

Integrate  SEP  with  integrated  master  plan. 

4.5.1 

3.4.5 

Integrate  SEP  with  integrated  master  schedule. 

4.5.1 

3.4.5 

Integrate  SEP  with  technical  performance  measures  (TPM). 

4.5.1 

3.4.4 

Integrate  SEP  with  earned  value  management. 

4.5.1 

3.4.5 

Use  TPMs  to  compare  actual  versus  planned  technical 
development  and  design  maturity. 

4.5.5 

3.4.4 

Use  TPMs  to  report  degree  to  which  system  requirements  are 
met  in  terms  of  performance,  cost,  and  schedule. 

4.5.5 

3.4.4 

Use  standards  and  models  to  apply  systems  engineering. 

4.2.2, 

4.2.2. 1 

Institute  requirements  management  and  traceability. 

4. 2. 3. 4 

3.4.4 

Use  EVM. 

11.3.1 
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Success  Criteria 

The  standards  discuss  the  importance  of 
holding  technical  reviews  at  various  stages 
of  development  to  assure  that  all  success 
criteria  have  been  met.  IEEE  1220  pro¬ 
vides  success  criteria  to  be  used  at  major 
technical  reviews.  For  example,  some  of 
the  success  criteria  for  a  preliminary 
design  review  are  the  following: 

•  Prior  completion  of  subsystem 
reviews. 

•  Determine  whether  total  system 
approach  to  detailed  design  satisfies 
tire  system  baseline. 

•  Unacceptable  risks  are  mitigated. 

•  Issues  for  all  subsystems,  products, 
and  life-cycle  processes  are  resolved. 
The  success  criteria  should  be  defined 

in  a  SEP  or  other  technical  plan.  The  cus¬ 
tomer  should  review  this  plan  with  the 
supplier  and  reach  agreement  on  the  suc¬ 
cess  criteria  to  be  used  at  technical 
reviews. 

Technical  Performance 
Measurement 

Technical  performance  measurements 
(TPMs)  are  defined  and  evaluated  to 
assess  how  well  a  system  is  achieving  its 
performance  requirements.  TPM  uses 
actual  or  predicted  values  from  engineer¬ 
ing  measurements,  tests,  experiments,  or 
prototypes.  IEEE  1220,  EIA  632  and  “A 
Guide  to  the  Project  Management  Body 
of  Knowledge”  (PMBOK  Guide)  [7]  pro¬ 
vide  similar  guidance  for  TPM  planning 
and  measurement  and  for  integrating 
TPM  with  EVM.  For  example,  EIA  632 
states  that  TPMs  predict  tire  future  value 
of  key  technical  parameters  of  the  end- 
system  based  on  current  assessments  and 
that  milestones  are  established  for  com¬ 
paring  planned  and  actual  progress. 

SE  Work  Products 

The  SE  process  generates  significant  work 
products  that  should  be  included  in  inte¬ 
grated  planning  and  measured  with  EV. 
The  process  products  of  IEEE  1220  are 
as  follows: 

•  Requirements  baseline. 

•  Validated  requirements  baseline. 

•  Functional  architecture. 

•  Verified  functional  architecture. 

•  Physical  architecture. 

•  Verified  physical  architecture. 

These,  or  similar,  work  products 

should  be  included  in  tire  integrated  mas¬ 
ter  schedule,  be  the  output  of  work  pack¬ 
ages,  and  have  defined  success  criteria. 

CMM  Integration 

The  CMM  Integration™  (CMMI®)  [8]  pro¬ 


vides  many  practices  that  augment  the 
EVMS  guidelines.  CMMI  also  lists  typical 
work  products  (TWPs)  within  process 
areas.  To  ensure  traceability  of  product 
quality  requirements  to  work  tasks  and 
work  products,  these  TWPs,  or  similar 
artifacts,  should  be  the  outcome  of  work 
packages.  Here  are  some  TWPs  in  CMMI. 

Requirements  development  TWPs 
include  the  following: 

•  Derived  requirements. 

•  Product  requirements. 

•  Product-component  requirements. 

•  Interface  requirements. 

•  Activities  diagrams  and  use  cases. 

•  Results  of  requirements  validation. 
Technical  solution  TWPs  include  the 

following: 

•  Documented  relationships  between 
requirements  and  product  compo¬ 
nents. 

“The  distinguishing 
feature  of  PB>EV  is  its 
focus  on  the  customer 
requirements  ...  Progress 
is  measured  against  a 
plan  to  fulfill  all 
customer  requirements  ... 
management  is  able  to 
take  rapid  corrective 
actions  on  deviations 
that  threaten  customer 
satisfaction  ...” 


•  Product-component  designs. 

•  Technical  data  packages. 

•  Allocated  requirements. 

•  Verification  criteria  used  to  ensure 
requirements  have  been  achieved. 

•  Interface  control  documents. 

•  Implemented  design. 

Verification  TWPs  include  these: 

•  Exit  and  entry  criteria  for  work  prod¬ 
ucts. 

•  Verification  results. 

A  decision  analysis  and  resolution 
TWP  includes  tire  results  of  evaluating 
alternate  solutions. 

Cost  Savings 

Measurement  costs  money.  An  enterprise 


must  incur  significant  implementation  and 
sustainment  costs  to  use  EVM.  These 
costs  can  be  reduced  if  the  enterprise  uti¬ 
lizes  an  effective  process  to  determine 
what  needs  to  be  measured  and  limits  the 
measurements  to  those  that  meet  its  infor¬ 
mation  needs  and  objectives.  Further¬ 
more,  management  can  be  more  effective 
if  it  focuses  on  fewer  but  more  critical 
measures. 

PBEV  is  cost-effective  because  it  lim¬ 
its  the  number  of  activities  that  should  be 
discretely  measured  to  those  that  meet 
defined  information  needs  such  as  the 
work  products  described  above.  Other 
measurable  activities  may  be  planned  as 
level  of  effort,  if  it  is  not  practicable  to 
measure  them,  or  they  may  be  appor¬ 
tioned  effort.  Additional  measurement 
guidance  is  available  in  a  Software 
Engineering  Institute  technical  note  [9], 

PBEV  Characteristics 

PBEV  is  a  set  of  principles  and  guide¬ 
lines  that  specify  the  most  effective  meas¬ 
ures  of  cost,  schedule,  and  product  qual¬ 
ity  performance.  It  has  several  character¬ 
istics  that  distinguish  it  from  traditional 
EVMS: 

•  Plan  is  driven  by  product  quality 
requirements,  not  work  requirements. 

•  Focuses  on  technical  maturity  and 
quality,  in  addition  to  work. 

•  Focuses  on  progress  toward  meeting 
success  criteria  of  technical  reviews. 

•  Adheres  to  standards  and  models  for 
SE,  software  engineering,  and  project 
management. 

•  Provides  smart  work  package  plan¬ 
ning. 

•  Enables  insightful  variance  analysis. 

•  Ensures  a  lean  and  cost-effective 
approach. 

•  Enables  scalable  scope  and  complexity 
depending  on  risk. 

•  Integrates  risk  management  activities 
with  the  performance  measurement 
baseline. 

•  Integrates  risk  management  outcomes 
with  the  Estimate  at  Completion. 

PBEV  augments  EVMS  with  four  addi¬ 
tional  principles  and  16  additional  guide¬ 
lines. 

PBEV  Principles 

The  following  are  PBEV  principles  that 
set  it  apart  from  EVMS: 

1.  Product  Scope  and  Quality.  Inte¬ 
grate  product  scope  and  quality 
requirements  into  the  performance 
measurement  baseline. 

2.  Product  Quality  Requirements. 
Specify  performance  toward  satisfying 
product  quality  requirements  as  a  base 
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measure  of  earned  value. 

3.  Risk  Management  Integration.  In 

tegrate  risk  management  with  EVM. 

4.  Tailor  PBEV.  Tailor  the  application  of 
PBEV  according  to  die  risk. 

The  first  two  PBEV  principals  are  dis¬ 
cussed  below  in  greater  detail. 

Product  Scope  and  Quality 

The  first  principle  introduces  two  control 
elements  that  distinguish  PBEV  from 
EVMS:  product  scope  and  product  quali¬ 
ty  requirements.  This  principle  focuses  on 
customer  satisfaction,  which  is  based  on 
delivery  of  a  product  that  meets  its  quali¬ 
ty  requirements  and  is  within  the  cost  and 
schedule  objectives.  The  supplier  has 
business  objectives  to  achieve  maximum 
customer  satisfaction  and  to  deliver  the 
product  with  the  best  possible  cost  per¬ 
formance. 

Product  Quality  Requirements 

In  the  context  of  PBEV,  the  product 
scope  is  defined  and  bounded  in  terms  of 
product  quality  requirements.  A  product 
quality  requirement  is  a  characteristic  of  a 
product  that  is  mandatory  in  order  for  the 
product  to  meet  verified  customer  needs. 
The  set  of  product  quality  requirements 
becomes  the  product  requirements  base¬ 
line  that  is  integrated  into  die  perform¬ 
ance  measurement  baseline  along  with 
work  scope,  schedule,  and  cost  objectives. 

Product  quality  is  also  discussed  in 
El  A  632  (Requirement  10): 

a.  Identify  product  metrics,  and  their 
expected  values,  that  will  affect  the 
quality  of  the  product  and  provide 
information  of  the  progress  toward 
satisfying  the  acquirer  and  other  stake¬ 
holder  requirements,  as  well  as  derived 
requirements. 

b.  Compare  results  against  requirements 
to  determine  degree  of  technical  re¬ 
quirement  satisfaction,  progress  toward 
maturity  of  tire  system  (or  portion 
thereof)  being  engineered,  and  varia¬ 
tions  and  variances  from  requirements. 

Measuring  Quality 

Project  management  processes  require 
progress  reporting  at  periodic  intervals, 
normally  monthly.  However,  progress 
toward  achieving  product  quality  objec¬ 
tives  is  not  always  measurable  on  a  period¬ 
ic  basis.  For  example,  a  hardware  or  soft¬ 
ware  component  may  require  die  comple¬ 
tion  and  assembly  of  many  enabling  work 
products  such  as  drawings  or  coded  soft¬ 
ware  modules,  before  the  integrated  set  of 
work  products  may  be  measured  against 
product  quality  objectives.  Consequently, 
interim  progress  measurement  is  normally 


against  die  scheduled  completion  of  inter¬ 
mediate,  enabling  work  products. 

The  completion  criterion  for  an 
enabling  work  product,  such  as  a  drawing, 
is  determined  by  the  organization’s 
process  quality  procedures  and  standards. 
Successful  peer  reviews  or  testing  are 
often  used  to  determine  the  completeness 
of  interim  work  products  against  process 
quality  procedures. 

PBEV  provides  guidance  to  measure 
performance  toward  achieving  a  combina¬ 
tion  of  the  following: 

•  Schedule  objectives  for  enabling  work 
products  that  meet  process  quality 
objectives. 

•  Event-driven  quality  objectives  when 
die  event  is  the  achievement  of  meas¬ 
urable  product  quality  requirements. 
Also,  the  achievement  of  significant 

performance  requirements  may  not  be 
measurable  at  the  component  or  subcom¬ 
ponent  level  but  may  depend  on  achieving 
planned  TPM  or  other  quality  objectives 
that  are  measurable  at  higher  levels  of  the 
system  architecture.  Consequently,  EV  at 
die  work  package  level  may  be  quantita¬ 
tively  linked  to  the  performance  of  inte¬ 
grated  components  at  a  higher  level  of  the 
work  breakdown  structure. 

During  the  system  definition  stage,  and 
with  the  evolutionary  acquisition  approach, 
die  real  product  quality  requirements  are 
not  yet  known  [10].  Consequendy,  activity 
accomplishment  criteria  should  be  estab¬ 
lished  to  determine  progress  assessment 
until  die  early  product  quality  requirements 
have  been  determined. 

Evolutionary  Acquisition 

Per  the  DAG,  when  a  program  uses  an 
evolutionary  strategy,  each  development 
increment  should  have  a  specific  set  of 
parameters  with  thresholds  and  objectives 
appropriate  to  the  increment  (DAG, 
Section  2.3.2).  Within  the  development 
increment,  trade  studies  are  used  to 
resolve  conflicts  between  operational 
capabilities,  and  functional  and  perform¬ 
ance  requirements  (Section  4.5.6). 

PBEV  supports  evolutionary  acquisi¬ 
tion  because  it  is  based  on  requirements, 
both  those  diat  are  known  by  the  end  of 
a  development  increment  and  those  that 
are  evolving  during  die  increment.  The 
work  products  specified  in  PBEV 
Guideline  2.2  include  trade  study  data  to 
substantiate  that  system  requirements  are 
achievable. 

PBEV  Guidelines 

The  PBEV  guidelines  are  listed  in  Table  2 
with  references  to  their  source  standards 
and  models. 


Application  at  Northrop 
Grumman 

PBEV  began  with  a  series  of  process 
improvements  at  Northrop  Grumman 
Integrated  Systems.  The  company  was 
driven  by  die  need  to  improve  software 
development  measurement.  Initial  im¬ 
provements  were  based  on  Practical 
Software  and  Systems  Measurement 
(PSM)  [11],  Examples  of  performance- 
based  measures  for  EV  from  PSM  include 
functional  requirements  status,  compo¬ 
nent  status,  test  status,  and  increment  con¬ 
tent-function. 

A  previous  CROSSTALK  article, 
“Practical  Software  Measurement,  Perfor¬ 
mance-Based  Earned  Value,”  discusses 
lessons  learned,  the  improvement  process, 
and  provides  examples  of  the  types  of 
measures  that  were  discarded  and  imple¬ 
mented  [12],  i.e.,  the  measurement  of  de¬ 
fects  was  retained  as  an  indicator  of  qual¬ 
ity  and  a  predictor  of  final  cost  and  sched¬ 
ule.  However,  various  measures  of  achiev¬ 
ed  requirements  were  used  for  schedule 
progress  and  EV  instead  of  defect 
removal.  Also  provided  is  advice  regarding 
the  suitability  of  measuring  source  lines  of 
code,  defect  and  rework  planning,  and 
accounting  for  deferred  functionality. 
Many  of  these  techniques  have  been 
incorporated  into  die  NAVAIR  handbook, 
“Using  Software  Metrics  &  Measurement 
for  Earned  Value  Toolkit”  [13], 

These  improvements  paid  off  during 
upgrades  of  the  B-2  weapon  system.  The 
new  measures  helped  to  make  it  a  very 
successful  program. 

The  B-2  Spirit  Stealth  Bomber 
Program  implemented  several 
innovative  process  improvements 
using  EVM.  These  include  inte¬ 
grating  earned  value  with  systems 
engineering  processes,  defining 
improved  software  engineering 
metrics  to  support  EVM,  and 
developing  a  leaner,  more  effective 
methodology  called  Performance- 
Based  Earned  Value  [PBEV].  The 
PBEV  methodology  was  used  to 
ensure  diat  the  warfighter  received 
the  most  functionality  from  soft¬ 
ware  development  efforts.  On 
Joint  Standoff  Weapon/ Generic 
Weapon  Interface  System,  we  pro¬ 
vided  85  percent  more  capability 
than  originally  planned,  on  sched¬ 
ule  and  under  budget.  [14] 

Process  improvement  at  the  sector  is 
ongoing.  Current  policy  requires  alignment  of 
sector  processes  with  IEEE  1220  and  a 
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Performance-Based  Earned  Value 


Performance-Based  Earned  Value  Guidelines 

Source 

Section  Number 

1.1  Establish  product  quality  requirements  and  allocate  these  to  product 
components. 

CMMI® 

PMBOK  Guide 

RD  SP  2.1, 2.2 

8.1. 1.3 

1 .2  Maintain  bidirectional  traceability  of  product  and  product  component  quality 
requirements  among  the  project  plans,  work  packages,  planning  packages, 
and  work  products. 

CMMI 

PMBOK  Guide 

RM  SP  1.4 

5.5 

1 .3  Identify  changes  that  need  to  be  made  to  the  project  plans,  work  packages, 
planning  packages,  and  work  products  resulting  from  changes  to  the 
products  quality  requirements. 

CMMI 

PMBOK  Guide 

RM  SP  1.5 

4.3,  5 

2.1  Define  the  information  need  and  objective  to  measure  progress  toward 
satisfying  product  quality  requirements. 

CMMI 

IEEE  1220 

EIA  632 

PMBOK  Guide 

MA  SP  1.1 

6. 8. 1.5,  6.8.6 

4.2.1, 4.2.2 

5. 2. 3.1,  5.5, 

8. 1.3. 5 

2.2  Specify  work  products  and  performance-based  measures  of  progress  for 
satisfying  product  quality  requirements  as  base  measures  of  earned  value. 
Examples  are  the  following: 

•  Results  of  trade-off  analysis. 

•  Allocated  requirements  developed,  implemented  into  design, 
or  tested  successfully. 

•  Achieving  planned  TPMs. 

•  Meeting  entry  and  success  criteria  for  technical  reviews. 

•  Other  quality  objectives  achieved. 

CMMI 

CMMI 

CMMI 

IEEE  1220 

EIA  632 

PMBOK  Guide 

MA  SP  1.2 

RD  SP  3.3 

DAR  SP  1.5 
6.1.1.13,  6.7.6 

6. 8. 1.5,  6.8.6 

4.2.1, 4.2.2, 

4.5.1 

5. 2. 3.1, 8.2. 1.4, 

8. 1.3. 5, 

10.3.1.5, 

Glossary 

2.3  Specify  operational  definitions  for  the  base  measures  of  earned  value, 
stated  in  precise,  unambiguous  terms  that  address: 

•  Communication:  What  has  been  measured,  how  was  it  measured, 
what  are  the  units  of  measure,  and  what  has  been  included  or  excluded? 

•  Repeatability:  Can  the  measurement  be  repeated  given  the  same 
definition  to  get  the  same  results? 

CMMI 

PMBOK  Guide 

MA  SP  1.2 

8. 1.3. 2 

2.4  Identify  event-based  success  criteria  for  technical  reviews  that  include 
development  maturity  to  date  and  the  product's  ability  to  satisfy  product 
quality  requirements. 

IEEE  1220 

EIA  632 

3. 1.1. 6,  4.12, 

5.2.4,  5.3.4,  6.4, 

6.6,  6. 8. 1.5 

4.2.2 

2.5  Establish  time-phased  planned  values  for  measures  of  progress  toward 
meeting  product  quality  requirements,  dates  of  frequency  for  checking 
progress,  and  dates  when  full  conformance  will  be  met. 

IEEE  1220 

EIA  632 

PMBOK  Guide 

6. 8. 1.5,  6.8.6, 

4.2.1, 4.2.2, 
Glossary 

11.6.2.4 

2.6  Allocate  budget  in  discrete  work  packages  to  measures  of  progress  toward 
meeting  product  quality  requirements. 

IEEE  1220 

EIA  632 

PMBOK  Guide 

6. 8. 1.5,  6.8.6 

4.2.1 

5. 2. 3.1,  10.3.1.5 

2.7  Compare  the  amount  of  planned  budget  and  the  amount  of  budget  earned 
for  achieving  progress  toward  meeting  product  quality  requirements. 

IEEE  1220 

EIA  632 

PMBOK  Guide 

6. 8. 1.5,  6.8.6 

4.2.2,  6. 1.2. 6 
11.6.2.3 

2.8  Use  Level  of  Effort  method  to  plan  work  that  is  measurable,  but  is  not  a 
measure  of  progress  toward  satisfying  product  quality  requirements,  final 
cost  objectives,  or  final  schedule  objectives. 

CMMI 

LL 

MA  SP  1.2 

2.9  Perform  more  effective  variance  analysis  by  segregating  discrete  effort  from 
Level  of  Effort. 

LL 

3.1  Identify  changes  that  need  to  be  made  to  the  project  plans,  work  packages, 
planning  packages,  and  work  products  resulting  from  responses  to  risks. 

PMBOK  Guide 

11.1.3,  11.6.3.2 

3.2  Develop  revised  estimates  of  costs  at  completion  based  on  risk  quantification. 

PMBOK  Guide 

7. 3. 2. 3 

4.1  Apply  PBEV  coverage  to  the  whole  work  breakdown  structure  or  just  to  the 
higher  risk  components. 

CMMI 

LL 

MA  SP  1.2 

4.2  Apply  PBEV  throughout  the  whole  system  development  life  cycle  or  initiate 
after  requirements  development. 

CMMI 

LL 

MA  SP  1.2 

.  _  .  _ ,  ,  .  ..  ©  2005  Paul  Solomon 

Key  to  Abbreviations 

RD:  Requirements  Development  Process  Area  SP:  Specific  Practice 

RM:  Requirements  Management  Process  Area  MA:  Measurement  and  Analysis  Process  Area 

DAR:  Decision  Analysis  and  Resolution  Process  Area  LL:  Author's  Lessons  Learned  and  Process  Improvements 
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process  architecture  that  is  CMMI-compliant. 

Agile  Methods 

PMs  have  begun  to  use  agile  development 
methods  to  streamline  the  acquisition 
process.  Alistair  Cockburn  stated  that 
being  agile  is  a  declaration  of  prioritizing 
for  project  maneuverability  with  respect  to 
shifting  requirements,  shifting  technology, 
and  a  shifting  understanding  of  die  situa¬ 
tion  [15].  He  also  discusses  an  agile 
approach  to  using  earned  value  with  burn- 
down  charts  where  the  requirements 
change  frequently  [16]. 

However,  using  agile  acquisition 
streamlining  does  not  justify  the  elimina¬ 
tion  of  key  program  documents  and  solid 
program  planning.  Blaise  Durante,  the 
U.S.  Air  Force  deputy  assistant  secretary 
for  Acquisition  Integration,  stated  that 
implementing  Agile  Acquisition  requires 
die  following  [17]: 

•  Using  innovative  thought. 

•  Flexibility. 

•  Focusing  on  outcomes  versus  non¬ 
value-added  processes  and  reviews. 

•  Empowering  program  managers  to 
use  the  system  versus  being  hampered 
by  over-staff  management. 

•  Going  back  to  the  basics  of  program 
management. 

PBEV  can  support  agile  systems 
development.  Because  it  uses  require- 
ments-based  planning  and  performance- 
based  measurement,  it  enables  innovation, 
flexibility,  and  focusing  on  outcomes 
instead  of  non-value-adding  processes. 
Also,  PBEV  Guidelines  4.1  and  4.2  sup¬ 
port  agility  by  tailoring  the  application  of 
PBEV.  Discrete  measurement  may  be 
applied  only  to  the  higher  risk  compo¬ 
nents  of  the  WBS  and  may  be  deferred 
until  die  initial  requirements  have  been 
developed. 

Conclusions 

PBEV  supplements  traditional  EVMS  widi 
die  best  practices  of  SE,  software  engi¬ 
neering,  and  project  management  stan¬ 
dards  and  models.  Its  principles  and  guide¬ 
lines  enable  true  integration  of  project 
cost,  schedule,  and  technical  performance. 

The  distinguishing  feature  of  PBEV  is 
its  focus  on  die  customer  requirements. 
Measures  of  product  scope  and  product 
quality  are  incorporated  into  the  project 
plan.  Progress  is  measured  against  a  plan  to 
fulfill  all  customer  requirements.  Measuring 
die  wrong  things  does  not  dilute  manage¬ 
ment  attention.  Consequently,  manage¬ 
ment  is  able  to  take  rapid  corrective  actions 
on  deviations  diat  direaten  customer  satis¬ 
faction  and  business  enterprise  objectives. 
PBEV  also  integrates  risk  management 


widi  EVM.  Finally,  because  it  is  scalable, 
risk-based,  and  responsive  to  changing  cus¬ 
tomer  requirements,  PBEV  can  support 
evolutionary  acquisition  and  agile  systems 
development. 

It  is  recommended  that  process 
improvement  programs  include  plans  to 
incorporate  PBEV  principles  and  guide¬ 
lines.  ♦ 
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Balanced  Scorecards:  From  Golf  to  Business 


Bill  Ravensberg 

Quality  Assurance  Analyst 

Metrics  are  all  around  us  and  we  use  them  every  day  —  even  if  we  don’t  realise  it.  Golf,  that  frustrating  game  that  we  love 
to  play,  is  used  as  an  analogy  to  help  better  relate  to  the  concepts  and  values  of  a  Balance  Scorecard;  or,  if  you  understand 
Balanced  Scorecards,  then  to  golf!  Any  metric  is  fine  by  itself  but  can  become  much  more  meaningful  and  useful  when  com¬ 
bined  with  others.  Learn  why  a  Balanced  Scorecard  is  an  appropriate  way  of  reporting  on  a  collection  of  metrics.  The  suc- 
cesful  implementation  and  application  of  scorecards  enables  the  use  of  appropriate  metrics  to  facilitate  understanding,  plan¬ 
ning,  and  communications. 


Winston  Churchill  once  noted,  “Golf 
is  a  game  whose  aim  is  to  hit  a  very 
small  ball  into  an  even  smaller  hole,  with 
weapons  singularly  ill-designed  for  the 
purpose  [1].” 

Given  that,  you  would  wonder  why 
anyone  would  want  to  play  such  a  game. 
Yet,  there  is  an  attraction  to  golf  for 
many  people,  including  me.  And,  not 
only  do  we  play  the  game,  but  we  also  do 
the  following: 

•  Keep  our  score  and  compare  it  to  a 
target  known  as  par. 

•  Keep  track  of  our  scores  and  establish 
a  personal  benchmark  known  as  a 
handicap. 

•  Compare  our  new  scores  to  the  bench¬ 
mark  and  adjust  it  when  necessary. 

•  Analyze  what  we  did  right. 

•  Analyze  what  we  could  do  to  improve. 
•  Like  to  talk  about  it  . . .  usually! 

•  Do  benchmark  studies  when  we  com¬ 
pare  our  scores  and  handicaps  with 
others  and  categorize  ourselves  in 
groupings  and  rankings. 

•  Chart  our  progress  as  well  as  how  we 
are  doing  compared  to  others. 

•  Create  sub-measures  such  as  putts  per 
round,  sand  saves,  driving  accuracy, 
and  driving  distance  to  help  us  under¬ 
stand  our  strengths  and  weaknesses 
and  to  identify  and  prioritize  where  we 
need  to  improve. 

If  we  are  really  good,  we  can  take  our 
statistics  and  records  to  businesses  to 
solicit  them  as  sponsors,  because  they  will 
want  to  support  us  and  be  associated  with 
us  when  we  go  on  The  Tour. 

And,  if  measuring  ourselves  is  not 
enough,  the  golf  equipment  manufactur¬ 
ers  have  done  a  lot  of  measurements  on 
their  equipment.  They  have  applied  their 
goals  and  targets  to  research  and  develop¬ 
ment  to  create  the  technical  advances  in 
today’s  equipment  that  golfers  enjoy  and 
benefit  from.  In  turn,  the  manufacturer 
advertises  how  successful  players  have 
been  using  their  equipment. 


Before  the  Balanced 
Scorecard 

More  than  25  years  ago,  I  wanted  to  get 
serious  about  improving  my  golf  game. 
Basically,  I  did  what  many  have  done.  I 
talked  with  my  buddies  and  tried  to  figure 
out  what  I  should  do. 

I  found  that  I  needed  to  understand 
where  I  was  losing  strokes.  How  many  putts 
was  I  taking  on  each  green?  How  many 

“Many  people  generally 
look  at  measures  in 
isolation.  The  whole 
picture  is  not  always 
taken  into  consideration. 

But  measurement, 
in  isolation,  is  taking  a 
chance  that  the  results 
will  even  be  realized.” 

penalty  strokes  was  I  taking,  and  why  was  I 
taking  them?  Was  I  saving  or  wasting  shots 
around  the  greens  and  from  the  bunkers? 

I  had  to  pay  attention  to  what  I  was 
doing  when  I  played.  I  made  mental  notes 
and  compared  the  results  from  game  to 
game.  I  soon  learned  that  the  strategy  I 
needed  was  not  on  any  one  specific  thing 
but  it  needed  to  be  several  things  together. 

I  now  needed  to  determine  how  I  would  go 
about  improving  the  various  parts  of  my 
game.  I  started  with  the  areas  I  perceived  to 
be  the  worst  and  tried  to  focus  on  each  of 
them.  Unfortunately,  perception  is  not 
always  a  great  way  to  go.  Neither  is  trying  to 
work  on  several  different  areas  at  the  same 
time  —  at  least  without  a  strategy. 


Fortunately,  I  usually  played  with  the  same 
guys  and  they  were  able  to  help  me  under¬ 
stand  where  I  needed  to  improve.  Having 
somewhat  analyzed  my  past  performances, 
I  now  needed  to  determine  what  I  was 
going  to  do  to  improve  these  areas  of  my 
game.  Especially  if  I  wanted  to  beat  my 
friends! 

Unfortunately,  I  was  not  in  a  position  to 
significantly  increase  my  golf  expenses.  So, 
the  option  of  taking  lessons  could  not  be 
considered.  I  did  subscribe  to  a  golf  maga¬ 
zine  whose  format  and  content  I  liked.  I 
was  also  able  to  get  some  tips  from  my  golf 
buddies  as  we  each  knew  some  different 
things  about  the  game.  One  of  them  was  a 
fairly  accomplished  golfer,  and  I  was  able  to 
get  a  lot  of  good  tips  from  him. 

Next,  I  needed  to  practice  what  I  was 
learning.  This  involved  some  work  at  the 
driving  range  and  trying  some  things  as  I 
played.  I  figured  that  I  would  mess  up  every 
now  and  then  anyway,  so  it  would  not  mat¬ 
ter  much  if  I  messed  up  trying  something 
new.  (Note:  I  do  not  recommend  taking  this 
approach  at  work!) 

Ultimately,  I  did  improve.  However,  it 
was  a  lot  of  trial  and  error  based  on  my  per¬ 
ceptions  of  what  I  was  actually  experienc¬ 
ing  and  doing  as  I  played  each  game. 

Like  Churchill’s  thoughts  on  golf  clubs 
being  ill-designed  tools ,  the  tools  and  process¬ 
es  I  used  were  ill-designed.  I  wish  there  had 
been  a  tool  that  I  could  have  used.  Taking  a 
strategic  approach  to  tracking  the  data  and 
reporting  it  would  have  made  perfect  sense. 

What  about  the  tools  we  use  in  our 
work?  Are  they  appropriately  designed  for 
use  in  achieving  the  desired  results? 

The  Balanced  Scorecard  -  A 
Useful  Tool 

According  to  BetterManagement.com: 

A  Scorecard  is  essentially  a  carefully 
selected  set  of  measures  derived 
from  an  organization’s  strategy.  It’s  a 
tool  for  leaders  to  communicate  to 
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employees  and  external  stakeholders 
the  outcomes  and  performance 
drivers  by  which  the  organization 
will  achieve  its  strategic  objectives. 
Therefore,  the  Scorecard  provides 
the  link  that  translates  strategy  into 
action  across  the  enterprise,  aligning 
long-term  strategy  with  coordinat¬ 
ed,  cohesive  business  activities.  [2] 

According  to  IT  Management. , 

The  balanced  scorecard  approach 
was  developed  by  Dr.  David  Norton 
and  Dr.  Robert  Kaplan  of  Harvard 
University  around  1990.  Under  this 
approach,  conventional  financial 
measures  are  augmented  by  addi¬ 
tional  measures  that  report  on  the 
learning  and  growth  perspective, 
and  the  financial  perspective. 
However,  because  companies  and 
products  vary,  one  of  the  challenges 
of  using  the  balanced  scorecard 
approach  is  selecting  the  appropri¬ 
ate  metrics  for  each  of  the  four  seg¬ 
ments.  [3] 

The  Balanced  Scorecard  contains  four 
segments:  financial  perspective,  internal 
business  process  perspective,  customer 
perspective,  and  innovation  and  learning 
perspective.  The  financial  perspective  con¬ 
tains  the  traditional  financial  measures.  Its 
underlying  mission  is  to  represent  positive 


financial  contributions  by  the  division  to 
achieving  our  clients’  business  goals.  Such 
measures  as  the  average  cost  to  produce  a 
unit  of  product  or  process  can  be  used  to 
determine  the  relative  value  of  the  contri¬ 
butions  by  the  division  or  organization. 

The  internal  business  process  segment  per¬ 
spective  contains  measures  of  how  the 
internal  processes  are  performing.  The 
mission  is  to  deliver  timely  and  effective 
services.  We  need  to  know  what  services 
and  processes,  internal  to  our  division,  we 
must  excel  at  to  satisfy  our  customers. 

The  customer  perspective  contains  meas¬ 
ures  pertaining  to  tilings  that  concern  the 
customer.  The  underlying  mission  is  to 
represent  how  the  division  is  doing  in 
areas  that  directly  affect  the  clients.  Client 
satisfaction  surveys  and  ratings  supporting 
responses  to  client  queries  and  problems 
can  be  effective  in  demonstrating  cus¬ 
tomer  support. 

The  innovation  and  learning  perspective 
measures  the  learning  and  growth  of  the 
area.  The  mission  is  to  develop  the  internal 
capabilities  to  learn,  innovate,  and  exploit 
future  opportunities.  Success  in  this  area 
means  we  have  developed  the  ability  to 
change  and  improve,  enabling  us  to  better 
support  the  customer. 

In  summary,  there  are  four  segments  of 
the  Balanced  Scorecard,  each  consisting  of 
a  collection  of  measures.  These  measures, 
or  metrics,  are  statistics  that  we  can  collect, 
report,  and  use  to  review,  evaluate,  and 


determine  appropriate  action(s). 

When  used  properly,  we  make  metrics  a 
tool  and  not  the  weapon  that  many  have 
come  to  fear.  Let  me  demonstrate  the  use 
of  metrics  in  a  Balanced  Scorecard  using 
my  golf  improvement  experience. 

Improvement  Through  Metrics 

I  wanted  to  improve  my  golf  scores.  I  was 
not  happy  with  what  could  have  been  con¬ 
sidered  my  average  score  metric  when  I  start¬ 
ed.  However,  saying  that  I  wanted  to 
improve  my  score  and  actually  doing  it 
were  two  different  things. 

I  needed  a  strategy.  I  needed  to  consid¬ 
er  several  different  things  as  part  of  my 
strategy.  The  sample  Balanced  Scorecard  in 
Table  1  shows  several  of  the  areas  men¬ 
tioned  earlier  listed  as  specific  measures  in 
the  four  different  segments.  It  sure  would 
have  been  useful  to  have  this  Balanced 
Scorecard  to  report  on  die  state  of  my  golf 
experience  and  on  each  measure  by  taking 
all  the  data  tracked  from  all  the  rounds  of 
golf  played. 

Note  that  in  Table  1,  your  scorecard 
may  have  additional  measures  in  a  perspec¬ 
tive,  thus  the  blank  rows.  In  an  automated 
tool,  it  is  beneficial  to  have  links  to  the  def¬ 
initions  of  the  metrics,  their  data  charts, 
and  the  actual  data  for  the  metrics.  Also, 
the  frequency  of  reporting  can  be  whatev¬ 
er  is  needed  to  be  effective;  it  does  not  have 
to  be  the  same  for  all  measures.  Some 
could  be  annual,  some  quarterly,  and  others 
monthly. 

The  status  for  each  item  under  the 
four  perspectives  is  determined  by  com¬ 
paring  the  actual  results  against  specific 
targets.  This  is  how  the  green,  yellow,  and 
red  status  results  are  determined.  Setting 
appropriate  targets,  or  goals,  is  an  impor¬ 
tant  requirement.  Note  that  the  targets 
need  to  be  appropriate!  Targets  help  in 
understanding  how  the  measure  is  per¬ 
forming  in  respect  to  its  internal  goals  and 
strategies.  If  you  want  something  like 
stretch  objectives,  then  I  would  suggest  cre¬ 
ating  a  second  status. 

As  you  can  see,  not  all  the  measures  in 
the  Internal  Business  Process  Perspective 
are  on  the  green.  Some  are  in  the  sand  (yellow 
status)  while  others  are  out  of  bounds  (red 
status).  These  results  matched  my  game  at 
one  time.  Using  my  previously  discussed 
approach,  I  would  have  tried  to  do  some¬ 
thing  in  each  of  these  areas  to  fix  my  game. 
However,  having  data  that  I  can  analyze  for 
trends  and  tendencies  can  prove  quite  use¬ 
ful.  The  data  could  actually  be  showing  me 
that  die  results  of  the  odrer  processes  are 
linked  to  driving  accuracy.  Perhaps  all  that  is 
needed  is  to  work  on,  and  improve,  the 
driving  accuracy.  Then,  all  the  others  may 


Table  1:  Sample  Balanced  Scorecard  Using  Golf  Metrics 


Financial  Perspective 

Status 

Date 

Average  Cost  per  Round  of  Golf 

GREEN 

August 

Number  of  Golf  Rounds  per  Month 

YELLOW 

August 

Monthly  Practice  and  Learning  Cost 

GREEN 

August 

Internal  Business  Process  Perspective 

Status 

Date 

Average  Score 

YELLOW 

August 

Sand  Saves 

RED 

August 

Driving  Distance 

YELLOW 

August 

Driving  Accuracy 

RED 

August 

Greens  in  Regulation 

YELLOW 

August 

Putts  per  Greens  in  Regulation 

GREEN 

August 

Penalty  Strokes  per  Round 

YELLOW 

August 

Customer  Perspective 

Status 

Date 

Golf  Buddies'  Satisfaction 

GREEN 

August 

Golf  Team  Satisfaction 

GREEN 

August 

Innovation  and  Learning  Perspective 

Status 

Date 

Number  of  Lessons  per  Month 

GREEN 

August 

Hours  per  Month  Practicing 

GREEN 

August 

Number  of  New  Tips  Learned  per  Month 

GREEN 

August 

GREEN:  On  the  Green  YELLOW:  In  the  Sand  RED:  Out  of  Bounds 
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inherit  better  results.  I  would  not  be  able  to 
determine  dais  if  I  did  not  have  all  the  his¬ 
torical  data  for  analysis.  This  daen  elimi¬ 
nates  any  perception  that  may  have  result¬ 
ed  in  inaccurate  and  wasted  actions. 

Anodaer  daing  to  consider  is  daat  all  the 
metrics  in  green  status  in  dae  Financial 
Perspective  and  Innovation  and  Learning 
Perspective  could  be  indicating  something, 
too.  You  may  remember  that  I  did  not  want 
to  increase  my  financial  obligation.  But, 
something  the  overall  metrics  are  showing 
is  daat  maybe  I  needed  to  reconsider  this. 
Perhaps  putting  more  into  lessons  and 
playing  more  would  help  bring  dae  process 
metrics  into  better  shape. 

Basically,  dais  demonstrates  daat  daere 
are  many  variables  we  need  to  consider, 
and  daat  many  factors  come  into  play  in 
determining  what  options  can  be  taken  to 
address  improvement.  Many  people  gener¬ 
ally  look  at  measures  in  isolation.  The 
whole  picture  is  not  always  taken  into  con¬ 
sideration.  But  measurement,  in  isolation, 
is  taking  a  chance  that  dae  results  will  even 
be  realized.  Would  you  want  just  one  of  dae 
subjects  in  your  performance  appraisal 
used?  If  it  is  one  daat  you  did  well  in,  well, 
daat  would  be  great!  But,  what  if  it  is  a  sub¬ 
ject  you  need  improving  in?  And,  what  if 
your  job  or  next  pay  raise  was  based  on  it? 

As  alluded  to  in  Capers  Jones’  overview 
on  dae  expanding  roles  of  function  point 
metrics  [3],  different  metrics  are  appropri¬ 
ate  for  different  companies.  We  cannot  all 
use  the  same  metrics.  We  have  to  evaluate 
our  strategies  and  then  determine  what 
metrics  are  needed.  And,  we  need  to  deter¬ 
mine  what  measures  work  best  togedaer  to 
represent  a  complete  picture. 

Business,  Communication,  and 
Measurement 

The  business  areas  of  our  companies 
understand  measurement.  They  will  do 
internal  and  external  evaluations  and  com¬ 
parisons  as  a  way  of  measuring  and  under¬ 
standing  daemselves,  their  products,  daeir 
competition,  and  their  competitions’  prod¬ 
ucts.  They  will  determine  the  differences 
and  translate  their  findings  into  improve¬ 
ments  in  efficiency  and  effectiveness  to 
help  gain  market  share. 

In  turn,  the  business  wants  to  see  an 
efficient  and  effective  Information  Services 
(IS)  division.  When  a  competitor  comes 
out  with  a  new  product,  our  business  needs 
to  be  responsive  to  market  changes.  The  IS 
division,  as  a  service  provider  to  dae  busi¬ 
ness  areas  and  company,  needs  to  be 
accountable  and  responsive  so  daat  daese 
needs  can  be  quickly  achieved. 

The  division  needs  to  be  able  to  com¬ 


municate  in  a  value-added  way  with  dae 
business.  Clarified  terms  and  applications 
of  those  terms  is  a  good  start.  A  measure¬ 
ment  program  can  supplement  and 
enhance  communication,  not  only  with  dae 
business,  but  also  widain  dae  division.  It  is 
important  to  ensure  consistency  widain 
each  division  and  across  the  organization 
to  ensure  dae  measures  contain  dae  same 
kinds  of  data  so  daat  we  can  have  an  apples- 
to-apples  comparison  and  roll  up  to  a  multi¬ 
division  strategic  view  of  all  the  data. 

To  be  effective  in  our  communications, 
we  need  to  measure  and  report  actual  per¬ 
formance.  Do  not  make  daings  up.  Be  hon¬ 
est.  We  need  to  demonstrate  to  business 
management  that  the  IS  division  is  man¬ 
aged  wida  a  fact-based  approach.  The  joint 
evaluation  of  performance  trends  versus 
established  goals  or  targets  is  an  objective 
and  non-emotional  way  to  evaluate  and 
communicate. 

Measurement  is  key  and  it  needs  to  be 
relative.  It  adds  meaning  and  value  when 
looked  at  on  the  whole  and  not  individual¬ 
ly.  It  must  supply  useful  information  for 
decision-making.  Without  measurement, 
how  will  we  ever  know  if  we  are  improving 
our  processes  and  deliverable? 

You  may  want  to  consider  using  multi¬ 
ple  scorecards.  For  example,  one  for 
development,  one  for  support,  one  for 
project  management,  and  one  for  infra¬ 
structure  support. 

The  Balanced  Scorecard  comple¬ 
ments  financial  measures  of  past 
performance  with  measures  of  the 
drivers  of  future  performance,  say 
Kaplan  and  Norton.  Of  course, 
die  old  dictum  still  holds  overall: 
You  cannot  control  what  you  can¬ 
not  measure.  Thus  die  Balanced 
Scorecard  was  a  new  concept 
mainly  designed  to  translate  a  com¬ 
pany’s  vision  and  actions  into  a 
consistent  set  of  measures.  [4] 

Effective  measurement  provides  key 
learning  opportunities  that  can  be  used  in 
our  efforts  toward  continual  improvement. 
We  can  use  die  findings  to  identify  success¬ 
ful  practices  and  build  on  diem.  We  can 
eliminate  unsuccessful  practices  and 
improve  those  that  just  are  not  working  die 
way  they  were  intended.  A  better  under¬ 
standing  and  appreciation  of  otiier  divisions 
can  also  be  achieved  as  the  development  of 
die  metrics  and  their  required  data  are 
worked  through,  reported  on,  and  acted  on. 

Measurement  Lessons  Learned 

The  strategies  supporting  the  mission  and 


vision  need  to  be  considered  to  gain  an 
understanding  of  what  is  needed  to  per¬ 
form  die  measurements.  This  will  help  in 
determining  the  data  required  to  fulfill  the 
measurements. 

There  are  many  sources  of  data  when 
you  are  dealing  widi  a  large  number  of 
metrics.  Some  of  the  sources  include 
accounting,  projects,  function  point  analy¬ 
sis,  and  surveys.  Some  data  such  as  func¬ 
tion  points  are  not  used  by  themselves  for 
any  one  measure  but  are  a  component  of 
several  measures.  Other  data  often  used 
widi  function  points  include  cost,  full¬ 
time-equivalent  effort,  elapsed  weeks,  and 
defects. 

Ensure  the  metric  definitions  are  com¬ 
plete.  Fully  express  each  definition,  includ¬ 
ing  the  required  data  source(s),  desired 
trends,  and  the  rationale  to  ensure  under¬ 
standing.  Even  with  complete  definitions, 
the  data  source(s)  and  formula(s)  may 
change  from  die  original  understanding  to 
achieve  the  definitions’  rationale  as  the 
metric  is  developed.  Involve  the  people 
needed  for  providing  and  using  die  data 
early  in  the  process  to  get  buy-in.  This 
gready  reduces  the  time  required  to  sell  the 
metrics. 

Over  time,  the  current  set  of  metrics 
may  no  longer  meet  the  needs.  All  the  dif¬ 
ferent  metrics  that  make  up  a  scorecard 
need  to  be  monitored  and  adjusted  as 
strategic  plans  change.  The  measurements 
need  to  be  current  and  useful.  More,  or 
different,  metrics  may  be  needed. 
Therefore,  a  regular  evaluation  of  the 
metrics  is  required. 

Data  from  outside  the  organization, 
known  as  industry  benchmarks,  can  be 
used  to  gain  an  understanding  of  die  per¬ 
formance  in  respect  to  odier  companies  in 
the  industry.  This  data  needs  to  be  similar 
to  your  own  data  to  ensure  an  apples-to-apples 
comparison.  There  are  different  vendors 
that  have  data  for  diis  use.  They  need  to  be 
evaluated  to  determine  which  one  will  be  an 
appropriate  source  of  data  for  your  needs. 

So, Who  Is  No.  I? 

Like  in  golf  and  its  manufacturers  and 
players,  the  IS  division  can  realize 
improvements  and  higher  satisfaction 
both  internally  and  with  the  customer. 
The  scorecard  facilitates  communication 
widiin  the  division  and  with  the  business 
organization  by  providing  balanced  meas¬ 
ures  with  supporting  data. 

In  the  July  2003  issue  of  Golf 
Magazine,  Greg  Norman  commented 
about  how  die  business  world  fascinates 
him.  He  said: 

That  is  a  wonderful  thing  diat  golf 
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has  given  me.  What  amazes  me  is 
that  some  of  these  businessmen 
have  told  me,  ‘Greg,  you  know 
what,  we’re  great  in  our  business, 
but  you  have  something  99.9  per¬ 
cent  of  the  people  in  this  world 
don’t  have.  You  know  what  it’s  like 
to  be  number  one.’  They  may  be 
great  CEOs,  but  they  don’t  know  if 
they’re  the  best  CEO  because  it 
can’t  be  measured.  [5] 

The  Balanced  Scorecard  can  be  a  very 

useful  tool  . . .  when  properly  used!^ 
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Web  Sites 


Oklahoma  City-Air 
Logistics  Center 

www.bringittotinker.com 
The  76th  Software  Maintenance  Group 
at  the  Oklahoma  City  -  Air  Logistics  Center 
is  a  leader  in  the  avionics  software  indus¬ 
try  that  understands  the  importance  of 
total  system  integration.  The  center  has  a 
proven  track  record  of  producing  soft¬ 
ware  on  time,  on  budget,  and  defect-free. 
Its  staff  of  software  professionals  and 
industry  partners  provides  the  expertise, 
software,  weapons,  interface,  and  aircraft 
systems  that  are  fully  integrated  to  ensure 
dependable  war-winning  capabilities. 
The  center’s  areas  of  expertise  include 
navigation,  radar,  weapons  and  system 
integration,  systems  engineering,  opera¬ 
tional  flight  software,  and  more. 

Ogden-Air  Logistics 
Center 

www.mas.hill.af.mil 

The  309th  Software  Maintenance  Group 
at  the  Ogden-Air  Logistics  Center  is  a 
recognized  world  leader  in  cradle-to- 
grave  systems  support,  encompassing 
hardware  engineering,  software  engineer¬ 
ing,  systems  engineering,  data  manage¬ 
ment,  consulting,  and  much  more.  The 
division  is  a  Software  Engineering 
Institute  Software  Capability  Maturity 


Model®  (CMM®)  Level  5  Organization 
with  Team  Software  Process™  engineers. 
Currently  the  division  is  transitioning  to 
CMM  Integration™,  which  integrates 
systems  engineering  practices  with  soft¬ 
ware  engineering  processes.  This  model 
more  closely  matches  the  complex  hard¬ 
ware,  software,  and  systems  products  and 
capabilities  representative  of  the  organi¬ 
zation’s  breadth  of  products  and  services. 

Warner  Robins-Air 
Logistics  Center 

https:/ /wwwmil.robins.af.mil 
The  402d  Software  Maintenance  Group 
at  the  Warner  Robins  Air  Logistics 
Center  provides  combat-ready  weapon 
systems,  equipment,  services,  and  sup¬ 
port  personnel  for  the  U.S.  Air  Force. 
The  center  is  a  leader  in  systems  engi¬ 
neering;  reliability,  maintainability,  and 
availability  engineering;  safety  engineer¬ 
ing;  human  factors  engineering;  ad¬ 
vanced  design  and  manufacturing  engi¬ 
neering;  and  logistics  engineering  sup¬ 
port.  The  center  has  worldwide  manage¬ 
ment  and  engineering  responsibility  for 
the  repair,  modification  and  overhaul  of 
the  F- 1 5  Eagle,  C-130  Hercules,  C-141 
Starlifter,  C-5  cargo  aircraft,  U-2  surveil¬ 
lance  aircraft,  all  Air  Force  missiles,  all 
Air  Force  helicopters,  and  more. 
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Bayonets  and  Deployment 


♦ 


BackTalk 


Ever  since  I  was  old  enough  to 
read,  I  have  been  fascinated  by 
the  Civil  War.  As  a  military  dependent 
growing  up,  I  was  lucky  to  travel  a  lot, 
which  allowed  me  to  continually  talk 
my  father  into  taking  detours  to  see 
Civil  War  sites.  While  I  was  actually 
born  outside  of  the  United  States  (my 
dad  was  stationed  in  Edinburgh, 
Scotland  after  World  War  II),  my  par¬ 
ents  are  from  Georgia.  Travels  in  the 
South  gave  me  a  very  Southern  per¬ 
spective  on  the  “War  of  Northern 
Aggression.”  I  was  lucky  enough  to 
visit  Andersonville,  Ga.,  site  of  the 
Civil  War  prison;  the  battlefields 
around  Richmond,  Va.;  and  Chatta¬ 
nooga,  Tenn.,  site  of  Lookout 
Mountain  and  the  “Battle  Above  the 
Clouds.” 

Although  I  am  an  avowed 
Southerner,  one  of  my  heroes  has 
always  been  Joshua  Chamberlain,  who 
was  an  academic,  but  had  a  strong 
urge  to  fight  to  save  the  Union.  He 
volunteered  to  fight,  and  was  soon 
made  a  lieutenant  colonel.  He  fought 
in  many  battles  and  by  July  1863,  he 
was  made  a  colonel.  On  July  2,  1863, 
Col.  Chamberlain  was  at  Gettysburg 
and  was  given  orders  to  defend  Little 
Round  Top,  an  important  position 
giving  a  commanding  view  of  the 
entire  battlefield.  His  actions  during 
this  key  engagement  held  the  Union’s 
position  and  significandy  contributed 
to  the  Union  victory  at  Gettysburg,  a 
battle  that  is  now  viewed  as  the  turn- 
ing  point  of  the  Civil  War. 

Several  years  ago,  I  finally  got  a 
chance  to  visit  Gettysburg  and  made 
it  a  point  to  park  my  car  near  a  rocky 
area  called  Devils  Den  and  walk 
through  the  Slaughter  Pen  up  towards 
Little  Round  Top.  On  this  area.  Col. 
Chamberlain,  running  low  on  ammu¬ 
nition,  ordered  his  men  of  the  20th 
Maine  to  attach  bayonets;  he  led  a 
bayonet  charge  downhill  against  the 
15th  Alabama,  driving  them  back.  As 
a  result  of  this  and  other  heroic 
actions,  Col.  Chamberlain  (who  would 
be  Brevet  Maj.  Gen.  Chamberlain  by 
the  war’s  end)  received  the  Medal  of 
Honor.  When  the  Civil  War  ended, 
Gen.  Grant  chose  Chamberlain  to 
receive  the  formal  surrender  of 


weapons  and  colors  on  April  12,  1865. 

How  in  the  world  does  this  fit  in 
with  this  issue’s  theme,  “Systems: 
Fielding  Capabilities?”  Col.  Cham¬ 
berlain  was  able  to  surprise  the  15th 
Alabama  with  a  bayonet  charge,  and 
this  element  of  surprise  allowed  them 
to  succeed.  The  tide  of  this  journal  is 
“Crosstalk,  The  Journal  of 
Defense  Software  Engineering.” 
Notice  the  software  engineering. 
Many  times,  members  of  the  software 
engineering  profession  are  a  bit  short¬ 
sighted  about  their  products.  It’s  not 
the  software  that  will  win  the  war  — 
it’s  the  systems  capability. 

Being  the  best  shot  in  the  world 
will  not  help  you  if  you  run  low  on 
ammunition.  However,  knowing  that  a 
backup  capability  exists  —  and  having 
the  training  to  deploy  and  use  it  —  will 
provide  you  with  a  winning  capability. 
Under  heavy  fire,  running  low  on 
ammo.  Col.  Chamberlain  was  able  to 
remember  that  a  backup  capability 
existed.  His  soldiers  had  the  training 
to  use  the  backup  system,  and  thus 
win  the  battle. 

It’s  the  same  way  in  the  software 
world:  we  need  to  be  able  to  meet  the 
entire  needs  of  our  users,  not  just 
stop  with  producing  stovepipe  soft¬ 
ware  products  that  are  unable  to 
adapt,  integrate,  and  survive.  As  soft¬ 
ware  engineers,  we  need  to  remember 
that  our  job  is  to  help  win  the  war,  not 
just  produce  the  software. 

Winning  is  not  just  about  produc¬ 
ing  a  workable  software  system.  It’s 
about  meeting  and  fielding  complex 
systems  that  have  the  capability  to 
meet  the  total  needs  of  our  users, 
including  contingency  and  emergency 
conditions.  It’s  about  the  management 
needed  to  see  potential  conditions 
and  prepare  software  systems  and 
capabilities  that  meet  all  of  our  users’ 
needs. 

There  should  be  no  such  thing  as 
an  “unexpected  need”  at  the  software 
level.  Requirements  engineering  needs 
to  be  accomplished  on  two  levels  — 
system  requirements  and  software 
requirements.  It  is  critical  to  under¬ 
stand  the  entire  system  requirements 
prior  to  starting  software  require¬ 
ments. 


In  many  cases,  software  design  and 
coding  starts  prior  to  complete  soft¬ 
ware  requirements.  It’s  not  the  “cor¬ 
rect”  thing  to  do,  but  sometimes,  it’s 
the  only  choice  you  have.  However,  it 
is  absolutely  critical  to  complete  sys¬ 
tem  requirements  prior  to  software 
design  and  coding.  Without  complete 
system  requirements,  you  can’t  envi¬ 
sion  the  role  that  the  software  is  going 
to  fulfill  in  the  complete  system.  If 
the  conceptual  design  or  “vision”  of 
the  system  is  incomplete  then  the  role 
of  software  will  probably  be  incom¬ 
plete  as  well. 

The  path  to  fielding  successful 
capabilities  is  to  make  sure  that  sys¬ 
tem  requirements  are  fully  thought 
out  before  assigning  system  capabili¬ 
ties  to  software  components.  This 
requires  planning  and  analysis.  System 
architects  need  to  be  motivated  to 
uncover  “elements  of  surprise”. 

Perhaps  a  litde  prodding  with  a 
bayonet  would  provide  useful  motiva¬ 
tion.  I’ve  certainly  considered  it  for  a 
few  select  co-workers  in  the  past! 

—  David  A.  Cook,  Ph.D. 

Senior  Research  Scientist 
The  AEgis  Technologies  Group,  Inc. 

dcook@aegistg.com 

Can  You  BACKTALK? 

Here  is  your  chance  to  make  your 
point,  even  if  it  is  a  bit  tongue-in- 
cheek,  without  your  boss  censoring 
your  writing.  In  addition  to  accepting 
articles  that  relate  to  software  engineer¬ 
ing  for  publication  in  CROSSTALK,  we 
also  accept  articles  for  the  BACKTALK 
column.  BACKTALK  articles  should 
provide  a  concise,  clever,  humorous, 
and  insightful  perspective  on  the  soft¬ 
ware  engineering  profession  or  indus¬ 
try  or  a  portion  of  it.  Your  BACKTALK 
article  should  be  entertaining  and 
clever  or  original  in  concept,  design,  or 
delivery.  The  length  should  not  exceed 
750  words. 

For  a  complete  author’s  packet 
detailing  how  to  submit  your 
BackTalk  article,  visit  our  Web  site  at 
<www.stsc.hill.af.mil>. 
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